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ABSTRACT 


This  research  examines  advances  in  aviation  technologies  that  allowed  the 
Apache  to  become  the  world’s  premier  attack  helicopter.  This  is  one  of  a  series  of 
research  pieces,  conducted  under  the  auspices  of  an  ongoing  research  effort  sponsored  by 
Headquarters  U.S.  Army  Material  Command.  The  U.S.  Army  Aviation  and  Missile 
Command  (AMCOM)  has  contracted  with  the  University  of  Alabama  in  Huntsville 
(UAH)  to  do  this  research.  After  all  of  the  research  is  completed,  the  principal 
investigators  at  UAH  and  Massachusetts  Institute  of  Technology  (MIT)  will  do  a 
crosscutting  analysis  across  all  the  systems  to  identify  key  factors  that  can  be  used  to 
guide  future  decision-making.  This  thesis  presents  answers  to  a  structured  set  of 
questions  that  address  issues  concerning  outside  influences,  technology  maturity  and 
program  management.  It  evaluates  the  role  of  development  and  test  strategies,  and 
whether  these  have  helped  to  create  a  functional  system.  The  research  methodology  is  a 
Case  Study,  a  limited  number  of  questionnaires  were  sent  to  key  personnel  intimately 
involved  with  the  program  development.  This  thesis  provides  the  reader  with  a  thorough 
understanding  of  how  the  history  of  Army  aviation  has  evolved  leading  to  the 
requirement  for  an  attack  helicopter  on  the  modern  battlefield.  The  emphasis  of  this 
document  is  to  follow  a  major  weapon  system  through  its  lifecycle  leading  to  successful 
deployment.  Lessons  learned  are  presented  in  a  clear  concise  manner  addressing  issues 
of  prime  concern  to  any  size  program. 
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I.  INTRODUCTION 


A.  PURPOSE 

This  research  examines  advances  in  aviation  technologies  that  allowed  the 
Apache  to  become  the  world’s  premier  attack  helicopter.  This  thesis  presents  answers  to 
a  structured  set  of  questions  that  address  issues  concerning  outside  influences,  technology 
maturity  and  program  management.  It  evaluates  the  role  of  development  and  test 
strategies,  and  whether  these  have  helped  to  create  a  functional  system.  The  research 
methodology  is  a  Case  Study.  The  research  findings  and  conclusions  are  primarily  based 
on  answers  to  questionnaires  that  were  filled  out  by  key  personnel  in  the  Government  and 
contractor  Apache  Program  Managers’  (PM)  offices.  When  required,  the  questionnaire 
results  are  supplemented  with  my  research  and  interviews  with  key  personnel.  I  will 
complete  a  master  questionnaire,  address  my  research  questions,  and  elicit  relevant 
information  to  augment  the  answers  to  build  the  case  study.  The  title  is:  Case  Study  of 
the  Development  of  the  Apache  Attack  Helicopter  (AH-64). 


B.  BACKGROUND 

A  long  history  of  trial  and  error  has  led  to  the  development  of  an  aircraft  that 
dominates  the  battlefield  like  no  other  system.  The  highly  integrated  subsystems  that 
makeup  the  AH-64  Apache  allow  it  to  evolve  to  fight  in  nearly  any  conditions.  However, 
the  transition  to  an  attack  helicopter  presence  in  the  Army  arsenal  was  not  trouble-free. 
Obstacles  to  development  of  the  Apache  included  service  rivalries,  funding  cuts  and 
cultural  misunderstandings.  There  are  several  key  areas  that  led  to  the  successful  fielding 
and  continued  use  of  aviation  firepower  on  the  modem  battlefield. 

C.  RESEARCH  QUESTIONS 
1.  Primary  Question: 

What  were  the  key  aspects  of  the  Apache  development  that  resulted  in  its 
successful  employment  in  Desert  Storm? 
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2.  Secondary  Questions 

a.  To  what  extent  was  there  user  support  and  funding  stability  during 
system  development? 

b.  To  what  extent  did  the  maturity  of  critical  technologies  being 
integrated  into  the  Apache  influence  the  development?  How  were  the 
organizations  that  had  developed  the  critical  technologies  involved 
during  system  development? 

c.  Did  the  test  strategy  adequately  evaluate  the  system  for  operational 
use? 

d.  How  effectively  were  teams  employed  during  development? 

e.  What  was  the  key  issue  that  the  PM  had  to  deal  with  during  the 
project? 

D.  SCOPE 

The  scope  of  the  thesis  includes  factors  that  influenced  Apache  development,  and 
explores  how  well  the  system  performed  in  Desert  Storm.  It  considers  the  critical 
technologies  and  how  they  were  effectively  implemented  in  this  system.  The  thesis 
explores  the  interrelationship  of  players  such  as  users,  government  and  contractor 
program  managers,  technology  developers  and  testers  and  how  they  interacted  in  carrying 
out  the  development,  production  and  fielding  of  the  system.  The  research  method  is  a 
case  study,  including  questionnaires,  interviews  and  supplemental  research. 


E.  METHODOLOGY 

The  methodology  used  in  this  research  consisted  of  the  following  steps: 

i.  Identified  key  Government  and  Contractor  Program  Management 
personnel  instrumental  to  the  development  process. 

ii.  Sent  questionnaires  to  key  personnel. 

iii.  Conducted  a  literature  search  of  books,  magazine  articles,  CD-ROM 
systems,  Government  reports,  internet-based  materials,  and  other 
library  information  resources. 

iv.  Conducted  interviews  with  key  personnel, 

v.  Analyzed  this  data. 
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vi.  Data  combined  to  form  a  single  integrated  master  survey. 

Results  from  all  survey  questionnaires  and  research  form  the  basis  of  the  final 
written  Case  Study. 


F.  ORGANIZATION 


Chapter  II:  BACKGROUND  History  of  Army  Aviation  as  it  relates  to  this  thesis 

Overview  of  the  Apache  Attack  Helicopter 


Chapter  III:  DATA 
Chapter  IV:  ANALYSIS 

Chapter  V:  CONCLUSIONS 


Present  survey  data  and  research  data 

Address  research  Questions 
Build  a  single  overall  survey 

Present  conclusions  and  recommendations 


G.  BENEFITS  OF  THE  STUDY 

This  research  studies  the  issues  and  relationships  associated  with  the  development 
of  the  Apache  Attack  Helicopter.  This  case  study  is  one  of  a  series  being  prepared  under 
an  ongoing  research  effort  sponsored  by  Headquarters  U.S.  Army  Material  Command. 
The  U.S.  Army  Aviation  and  Missile  Command  (AMCOM)  has  contracted  with  the 
University  of  Alabama  in  Huntsville  (UAH)  to  do  this  research.  After  all  of  the  Case 
Study  research  is  completed,  the  Principal  Investigators  at  UAH  and  Massachusetts 
Institute  of  Technology  (MIT)  will  do  a  crosscutting  analysis  across  all  the  systems  to 
identify  key  factors  that  can  be  used  to  guide  future  decision-making.  The  case  studies 
will  be  made  available  to  the  Defense  Acquisition  University  (DAU)  and  the  Naval 
Postgraduate  School  (NPS)  to  use  in  teaching  and  research. 
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II.  BACKGROUND 


Dateline:  January  17,  1991  —  The  largest  military  assault  since  D-Day  began 
rather  unceremoniously  as  eight  Apache  Attack  Helicopters  are  led  out  into  the  desert  by 
four  Air  Force  MH-53J  Pave  Low  Helicopters.  Dubbed  Task  Force  Normandy,  their 
mission,  to  open  the  door  into  Iraq,  will  signal  the  beginning  of  the  Gulf  War.  This 
mission  would  mark  yet  another  success  in  the  long  history  of  Army  aviation. 

Aviation  has  seen  amazing  evolutionary  transformations,  from  the  balloon 
outposts  during  the  Civil  War  to  the  “flying  tanks”  of  the  Gulf  War.  In  order  to  gain  a 
full  understanding  of  this  warfighting  system,  one  must  first  look  at  the  broad 
environment  in  which  it  operates.  This  chapter  will  look  at  how  the  history  of  Army 
aviation  has  evolved  and  the  nuances  that  have  grown  among  the  different  services  as 
well  as  those  within  the  Department  of  the  Army.  I  will  briefly  discuss  the  history  of 
Army  Aviation  and  the  socioeconomic  and  political  forces  that  it  has  evolved  around.  I 
will  then  lay  out  the  Apache  program  time  line,  followed  by  a  description  of  the  aircraft 
system  as  it  entered  Desert  Stonn. 


A.  HISTORY  OF  AVIATION 

1.  Balloon  Corps 

The  Union  Army  established  the  first  “aviation  unit”  in  the  1860’s  during  the 
Civil  War.  Dubbed  the  “Balloon  Corps  of  the  Army  of  the  Potomac”1,  they  used 
balloons  to  place  observers  above  the  battlefield  to  track  enemy  movement.  This  gave 
the  commander  a  distinct  advantage  in  this  war  of  positioning.  The  Balloon  Corps  was 
later  placed  under  the  Signal  Corps  for  the  remainder  of  the  war.  It  appears  that  there 
was  considerable  distrust  for  this  new  technology  and  the  men  who  risked  their  lives  to 
make  it  work.  The  Balloon  Corps  was  disbanded  shortly  after  the  end  of  the  war.  This 
marked  the  first  of  several  shortsighted  moves  by  the  top  brass  to  thwart  Army  Aviation 
in  favor  of  ground  troops. 
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Balloons  were  once  again  called  into  service  in  1898  during  the  Spanish 
American  War.  During  the  fighting,  the  first  U.S.  airman  was  shot  down  in  combat  as  his 
balloon  was  hit  by  enemy  fire.  Any  hint  of  future  Army  aviation  ended,  as  the  balloons 
once  again  disappeared  from  the  inventory  at  war’s  end. 

Balloons  have  been  used  by  the  military  since  the  turn  of  the  century.  They  have 
evolved  from  one-man  observation  posts  to  highly  sophisticated  surveillance  platfonns. 
Balloons  have  also  been  used  to  drop  ordnance  in  times  of  war.  The  use  of  balloons 
marked  the  unofficial  beginning  of  Army  aviation.  Throughout  the  early  years,  aviation 
remained  a  fairly  mundane  communication  asset  in  the  Signal  Corps.  That  was  all  to 
change  with  the  onset  of  World  War  I. 

2.  Aircraft 

On  December  17,  1903,  the  first  flight  of  a  “heavier  than  air”  craft  took  place  at 
Kitty  Hawk,  North  Carolina.  The  Wright  brothers  succeeded  where  many  had  failed  and 
thus  brought  the  world  a  little  bit  closer  together.  By  August  1907,  the  Army 
Aeronautical  Division  was  established  to  promote  the  use  of  aircraft  in  the  military.  In 
the  fall  of  1908,  the  Wright  brothers  build  a  “heavier  than  air  flying  machine”  in 
response  to  Signal  Corps  request  for  proposal  (RFP).  During  initial  flight  tests,  Lt. 
Selfridge  became  the  first  aviation  casualty,  and  the  pilot,  Orville  Wright  sustained  severe 
injuries,  as  the  plane  they  were  riding  in  fell  to  the  ground.  This  event  brought  those 
opposed  to  aviation  out  of  the  woodwork.  Wilbur  Wright  quickly  repaired  the  plane  and 
resumed  flight-testing.  He  successfully  demonstrated  that  the  craft  exceeded  all  Army 
requirements  and  Wright  Brothers  aircraft  soon  entered  military  service. 

As  the  fledgling  aviation  fleet  began  to  evolve,  a  daring  young  Russian  inventor, 
Igor  Sikorsky,  was  trying  to  prove  his  helicopter  design.  In  1909  Sikorsky  got  his  craft 
off  the  ground,  marking  the  first  flight  of  a  counter  rotating,  twin-bladed  helicopter.  ~  As 
WWI  approached,  Sikorsky  was  forced  to  concentrate  his  efforts  on  large  military 
aircraft.  By  1914,  he  had  created  a  four-engine  aircraft  capable  of  carrying  one  thousand 
pound  bombs  for  the  Russian  Army.  Meanwhile,  back  in  the  U.S.,  Congress  officially 
created  the  Aviation  Section  within  Signal  Corps  on  July  18,  1914.  Aircraft  were 
beginning  to  be  used  by  the  Allied  forces  in  the  war.  By  May  1918,  Congress  saw  the 

importance  of  aviation;  through  the  Overman  Act  they  formed  the  “Air  Service”.  This 
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removed  aviation  assets  from  the  Signal  Corps,  giving  aviators  more  control  over  their 
own  destiny.  However,  just  as  aviation  needed  most  to  increase  the  research  and 
development  of  this  new  technology,  the  war  ended  and  defense  funding  was  once  again 
severely  cut. 

As  civil  aviation  boomed  in  the  1920’s  and  1930’s,  Anny  aviation  tried  to  find 
itself.  In  1926  Congress  established  the  Army  Air  Corps.  The  Air  Corps  spent  the  next 
several  years  concentrating  on  large  bombers;  close  air  support  was  practically  ignored. 
Military  doctrine  at  the  time  was  that  the  next  war  would  be  fought  on  the  ground  and 
from  high  in  the  air,  and  that  air  power  was  best  used  beyond  the  range  of  artillery.4 
Meanwhile,  in  May  1941,  the  first  sustained  flight  of  a  Sikorsky  V-300  helicopter  took 
place.  This  aroused  the  Air  Corps  interest  in  helicopters  and  on  20  April  1942,  Sikorsky 
delivered  the  first  XR-4  helicopter  to  Anny.  The  R-4  was  the  first  mass  produced 
military  helicopter.  They  were  used  for  observation,  reconnaissance,  and  medical 
evacuation  missions.  An  Army  R-4B  was  the  first  perfonn  a  military  rescue  behind 
enemy  lines  on  April  25,  1944  in  Burma. ^  Between  1942  through  1946,  the  Army  Air 
Force  had  purchased  over  300  helicopters.  However,  combat  usage  of  this  unproven 
technology  remained  rather  limited. 

Considerable  changes  hit  the  military  when  on  9  March  1942,  Congress 
established  three  separate  and  coequal  commands:  Army  Ground  Forces,  Army  Air 
Forces,  and  Army  Service  Forces.  This  division  of  power  was  in  its  infancy  as  WW  II 
raged  on.  Then  in  the  1947  the  National  Defense  Act,  formally  established  the  Air  Force. 
The  military  chiefs  met  to  carve  up  their  missions.  The  Anny  was  forced  to  limit  their 
fixed  wing  assets  to  less  than  five  thousand  pounds,  while  the  Air  Force  would  provide 
the  necessary  close  air  support.  This  historic  event  forced  the  Army  to  develop  helicopter 
fleets  to  compensate  for  the  loss  of  its  fixed  wing  support. 

The  United  States  entered  Korea  with  nearly  the  same  sad  state  of  readiness  that 
they  took  into  WW  II.  The  services  had  suffered  from  years  of  neglect  at  the  hands  of 
“downsizing”  after  the  war.  The  Air  Force  was  mainly  equipped  to  fight  a  nuclear  war 
with  heavy  bombers.  Once  the  few  significant  targets  were  eliminated  in  Korea,  the 
bombers  had  little  impact.  Helicopter  use  was  delegated  to  search  and  rescue  missions  as 
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the  Army  did  their  part  from  the  air.  As  the  war  raged  on.  Army  H-13  helicopters,  first 
fielded  in  1951,  were  retrofitted  with  stretchers  on  their  landing  skids  to  transport  the 
growing  number  of  wounded  to  Mobile  Anny  Surgical  Hospitals  (MASH).  By  war’s 
end,  over  eighteen  thousand  wounded  had  been  transported  by  H-13s.6  The  civilian 
version  of  the  H-19  Chickasaw  was  the  world’s  first  transport  helicopter.  Built  by 
Sikorsky,  the  H-19  could  carry  six  litters  and  one  medical  attendant  during  Medevac 
missions.  With  seating  for  twelve,  the  Chickasaw  was  also  used  as  a  troop  transport, 
utility  carrier,  and  rescue  helicopter.7  The  success  of  the  H-13  and  H-19  in  Korea  helped 
the  Army  brass  see  the  importance  of  the  helicopter  on  the  future  battlefield. 

B.  ATTACK  HELICOPTERS: 

The  use  of  force  from  the  air  dates  back  to  the  Balloon  Corps  and  their  limited 
attempts  to  ann  aviators.  With  their  growing  fleet  of  large  aircraft,  the  Air  Force  quickly 
perfected  aerial  bombing  techniques.  The  Boeing  B-17  “Flying  Fortress”  ushered  in  the 
use  of  an  all  around  aerial  attack  with  its  various  crew  gun  mounts  and  the  ball-turret 
mounted  beneath  the  huge  slow  aircraft.  Fighter  aircraft  were  developed  to  help  protect 
the  bombers.  However,  close  air  support  was  left  largely  to  the  different  services  and 
usually  heroic  individual  efforts.  Backyard  trial  and  error  continued  throughout  WW  I 
and  WW  II  as  ingenious  aviators  and  mechanics  attempted  to  arm  their  aircraft  for  battle. 

The  Army  Ground  Forces  Board  at  Ft.  Bragg,  North  Carolina,  documented  the 
first  formal  test  of  an  armed  helicopter  on  December  14,  1945.  The  purpose  of  the  test 
was  to  determine  if  a  recoilless  rifle  could  be  mounted  on  a  helicopter  and  fired  in  flight. 
Test  results  show  that  when  fired,  the  backpressure  of  the  75mm  rifle  broke  the  Plexiglas 
windscreen  and  slightly  buckled  the  tail  cone  of  the  test  aircraft.  Lacking  an  adequate 
means  of  sighting  the  gun,  the  testing  was  halted.  Helicopter  armament  was  brought  to  a 
standstill  for  the  next  several  years  as  the  fledgling  helicopter  industry  grew. 
Meanwhile,  the  Air  Force  continued  to  concentrate  on  fixed  wing  assets  and  nuclear  war. 

The  Army  used  lessons  learned  from  the  Korean  conflict  to  boost  their  helicopter 
transport  fleet.  When  the  Army  entered  Vietnam,  the  need  for  close  air  support  quickly 
became  a  priority.  The  entire  helicopter  fleet  came  under  enemy  fire;  it  wasn’t  long 
before  the  need  for  aerial  defense  was  realized.  The  Anny  relied  on  it’s  aging  fleet  of 
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CH-2 1  Shawnee  tandem  rotor  helicopters  as  flying  trucks.  Dubbed  the  “flying  banana”, 
this  was  the  first  true  multi-mission  helicopter,  utilizing  wheels,  skis  or  floats  for 
different  terrains.  Shawnee  was  the  fourth  of  a  line  of  tandem  rotor  helicopters  designed 
by  Piasecki.  The  slow,  CH-2 1  ’s  were  sitting  ducks  for  enemy  fire,  one  was  even  rumored 

g 

to  be  brought  down  with  a  Viet  Cong  spear.  The  CH-21  ’s  were  soon  outfitted  with  guns 
in  the  doorways  and  on  the  skids.  Several  different  gun  experiments  took  place  in  the 
early  1960’s.  Some  Shawnees  were  equipped  with  movable  nose  guns.  The  Army  even 
attempted  to  mount  a  B-29  Superfortress  ball-turret  beneath  a  CH-21,  but  this  experiment 
was  quickly  discarded  as  the  forces  of  the  blast  damaged  the  test  aircraft.  The  Shawnee 
remained  the  workhorse  of  the  Army  through  the  early  years  of  Vietnam.  Use  of  the  CH- 
21  ended  with  the  arrival  of  the  UH-1  Huey  and  the  CH-47  Chinook  to  the  battlefield. 


Figure  1 .  Test  of  .50  cal  Guns  on  CH-2 1  Shawnee  (1962) 


Bell  Helicopter’s  UH-1  Iroquois  was  a  result  of  an  Army  request  for  a  general 
utility  helicopter.  Bell  began  development  of  the  prototype  in  1955  to  meet  the  Army 
specification.  The  Huey  as  it  was  called  after  it's  original  model  designation,  the  HU-1, 
was  essentially  a  stretched  Bell  model  47  Sioux  with  room  for  seven  troops  or  three 
stretchers  in  it's  cargo  compartment  behind  the  pilot.  As  Huey’s  entered  service  in 
Vietnam  they  were  first  anned  with  two  door  guns. 
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Figure  2.  Huey  Door  Mounted  Machine  Gun 
The  CH-47  Chinook  tandem  rotor  helicopter  was  developed  in  the  late  1950’s  in 
order  to  meet  increased  demand  for  an  all-weather  heavy  cargo  carrier.  The  YCH-47A 
made  its  initial  flight  on  21  September  1961  and  was  fielded  to  Vietnam  in  the  mid 
1960’s.9  In  an  experimental  project,  Boeing  Vertol  equipped  four  Chinooks  with  five 
machine  guns,  two  20  mm  cannons,  two  rocket  launchers  and  a  “chin-mounted”  grenade 
launcher.10  Designated  “Guns- A-Go-Go”  these  heavily  armored  aircraft,  each  with  a 
crew  of  eight,  entered  service  in  late  1965.  The  aircraft  proved  highly  effective  clearing 
landing  zones  and  in  assault  missions.  Each  aircraft  was  capable  of  carrying  a  ton  of 
expendable  munitions.  However,  they  were  difficult  to  maintain  and  following  a  number 
of  accidents,  the  effort  was  terminated  in  1967  with  the  introduction  of  the  AH-1  Cobra. 


AFT  GUNNER  WITH  7.62MM 
or  .50  CAL  MACHINE  GUN 


FLANK  GUNNERS  WITH  7.62mm 


GRENADE  LAUNCHER 


FIXED  FORWARD-FIRING 
WEAPONS  (1  EACH  SIDE) 


1 2.75-inch  ROCKET  POD  or  7  62mm 
GATLING  MACHINE  GUN  (1  EACH  SIDE) 


Figure  3.  Heavily  Armed  “Guns-a-Go-Go”  Chinook 
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As  the  war  raged  on  in  Vietnam,  the  Army  realized  the  need  to  control  it's  own 
close  air  support.  In  June  1963,  the  Army  issued  a  request  for  proposal  (RFP)  for  the 
Advanced  Aerial  Fire  Support  System  (AAFSS).  A  competition  pitted  the  traditional 
helicopter  builders  Sikorsky  and  Bell  versus  Lockheed,  a  newcomer  to  the  helicopter 
trade  but  with  considerable  fixed  wing  experience.  Bell  entered  a  scaled-down  version  of 
it's  Iroquois  Warrior.  The  other  competitor  was  the  Sikorsky  S-66.  The  Sikorsky  design 
had  a  rotorprop  tail  rotor  which  could  rotate  on  it's  axis  90°  to  act  both  as  an  anti-torque 
rotor  or  as  a  pusher,  thereby  transforming  the  S-66  into  a  compound  aircraft  in  cruising 
flight.  The  Lockheed  AH-56A  Cheyenne  won  the  competition. 

On  May  3,  1967,  the  first  prototype  YAH-56  Cheyenne  rolled  out  of  the 
Lockheed  facility.  The  futuristic  design  had  exceeded  Army  expectations.  The 
Cheyenne  had  a  single  rigid  four-bladed  main  rotor  and  anti-torque  tail  rotor,  and  a  three- 

bladed  pusher.  The 
radical  design  of  the 
Cheyenne  helped  it  to 
reach  an  astonishing 
speed  of  256  miles  per 
hour,  over  twice  the  top 
speed  of  a  UH-1.  The 
rigid-rotor  Cheyenne, 

with  a  crew  of  two,  had  a 
swiveling  gunner's 

station  linked  to  rotating 
belly  and  nose  turrets, 
and  a  laser  range-finder  tied  to  a  fire  control  computer.  It  was  armed  with  a  30mm 
automatic  gun  in  the  belly  turret  and  a  40mm  grenade  launcher  or  a  7.62mm  Gatling 
machine  gun  in  the  chin-turret,  TOWs,  and  2.75  inch  rocket  launchers.  The  turret  guns 
were  slaved  to  the  pilot’s  or  copilot’s  helmet  sight,  this  allowed  them  to  aim  and  fire  by 
simply  turning  their  head.  The  age  of  the  attack  helicopter  had  arrived.  However,  as 
requirements  creep  set  in,  the  Cheyenne  became  even  more  complex,  expensive  and 
worst  of  all,  behind  schedule. 
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The  Army  had  an  immediate  need  for  firepower  in  Vietnam  and  top  brass  were 
impatient  with  the  slow  progress  of  the  Cheyenne.  By  January  1965,  the  Army  released  a 
RFP  for  an  interim  Attack  Helicopter,  “escort  gunship”.  Three  systems  competed  for  the 
contract,  the  Sikorsky  Sea  King,  Kaman  Seasprite  and  Bell  Cobra.  Bell  won  the  flyoff 
and  by  October  1967,  the  first  Cobra  missions  were  flown  in  Vietnam.  As  the  world’s 
first  attack  helicopter,  the  Cobra’s  mission  was  direct  fire  support,  armed  escort  and 
reconnaissance.  It  was  armed  with  a  40  mm  grenade  launcher,  7.62  mm  “minigun”  and 
2.75-inch  rocket  launchers.  The  Viet  Cong  named  the  Cobra  “Whispering  Death”. 


Figure  4.  AH-1  Cobra  Gun  Ship 


Stateside  attention  turned  once  again  to  the  struggling  Cheyenne  program. 
Rollout  of  Lockheed’s  first  prototype  YAH-56  Cheyenne  took  place  on  May  3,  1967. 
The  Air  Force  saw  the  Cheyenne  as  a  threat  to  its  close  air,  anti-tank  mission.  Secretary 
of  the  Air  Force  Harold  Brown  ordered  the  development  of  the  A- 10  Warthog  to  meet 
that  need.  As  the  Cheyenne  continued  to  have  technical  problems,  the  Cobra  was  proving 
itself  in  battle.  The  Anny  soon  realized  that  they  would  not  win  a  turf  war  with  the  Air 
Force.  With  the  A-10  project  in  full  swing,  the  Army  decided  that  they  wanted  a  smaller, 
more  agile  Advanced  Attack  Helicopter  (AAH)  with  a  less  complicated  fire  control  and 
navigation  system.  The  Cheyenne  contract  was  terminated  in  May  1969.  Through  this 
period,  the  Army  continued  to  desire  fixed-wing  close  air  support  (CAS)  from  the  Air 
Force.  To  that  end,  it  was,  relatively  easy  for  the  two  services  to  agree  that  the  attack 
helicopter  did  not  perfonn  CAS.  Instead,  it  was  an  extension  of  organic  firepower,  and 
the  Air  Force  would  continue  to  provide  CAS  with  fixed-wing  aircraft.  The  two  services 
agreed  to  consider  the  two  types  of  aircraft  as  complementary  rather  than  duplicative. 
Since  that  time,  there  have  been  no  serious  disagreements  over  aviation  missions  and 
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functions  between  the  Army  and  the  Air  Force.  The  new  helicopter's  mission  would 
eventually  be  filled  by  the  AH-64  Apache  Attack  Helicopter. 

C.  APACHE  ATTACK  HELICOPTER 

The  McDonnell  Douglas  (formally  Hughes)  AH-64  Apache  is  a  twin-engine 
rotary  wing  aircraft,  designed  as  a  stable,  manned  aerial  weapon  system.  With  its  two 
pilots  and  sophisticated  computers,  the  Apache  is  capable  of  defeating  a  wide  range  of 
targets,  including  armored  vehicles.  It  is  capable  of  performing  missions,  day  or  night  in 
adverse  weather  conditions.  Combined  with  the  integrated  Target  Acquisition 
Designation  Sight  /  Pilot  Night  Vision  Sensor  (TADS/PNVS),  the  platform  provides  day 
and  night  acquisition  and  designation  of  targets  and  hand-off  capabilities  in  support  of 
Hellfire  and  other  guided  munitions.  Aircraft  armament  includes  the  Hellfire  anti-tank 
missile  system,  30mm  automatic  chain  gun  and  2.75”  rockets.  The  platform  has  a  full 
range  of  aircraft  survivability  equipment  with  the  ability  to  withstand  hits  from  rounds  up 
to  23mm  in  critical  areas.  Powered  by  two  General  Electric  gas  turbine  engines,  the 
Apache  can  cruise  at  an  airspeed  of  145  mph  with  a  flight  endurance  of  over  three  hours. 
The  AH-64  is  transportable  in  the  C-5,  C141  and  C-17.  The  Apache  Attack  Helicopter 
contributes  a  highly  mobile  and  effective  firepower  asset  to  the  anti-armor  capability  in 
the  field.  The  Apache  development  time  line  is  depicted  in  figure  6,  this  chart  was  built 
from  information  contained  in  a  number  of  sources11  12  lj. 


Figure  5.  Apache  AH-64 
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D.  DESERT  STORM 


In  the  early  morning  of  17  January  1991,  an  Army  aviator  fired  the  first  shot  of 
Operation  Desert  Storm  from  an  Apache  helicopter.  Within  a  few  minutes,  two  teams  of 
Apaches  totally  destroyed  two  Iraqi  air  defense  radar  stations,  paving  way  for  the  air  war 
over  Iraq. 


Figure  6.  Apache  Armaments 

During  the  100-hour  ground  war,  Army  attack  helicopters  played  their  most 
decisive  role  ever  in  combat.  Whatever  doubts  remained  regarding  combat  effectiveness 
of  attack  helicopters  were  quickly  dispelled.  In  addition  to  the  attack  role,  helicopters 
were  used  for  air  assault,  reconnaissance,  transportation,  combat  search  and  rescue,  and 
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observation.  Dozens  of  aviation  units  and  several  hundred  helicopters  of  all  types  took 
part  in  the  Gulf  War. 


Length 

58.17  ft 

Height 

15.24  ft 

Wing  Span 

17.15  ft 

Primary  Mission 
Gross  Weight 

15,075  lb 

Standard  Day 

Hot  Day 

Hover  In-Ground  Effect 

15,895  ft 

14,845  ft 

Hover  Out-of-Ground  Effect 

12,685  ft 

11,215  ft 

Vertical  Rate  of  Climb 

2,175  fpm 

2,050  fpm 

Maximum  Rate  of  Climb 

2,915  fpm 

2,890  fpm 

Maximum  Level  Flight  Speed 

150  kt 

153  kt 

Cruise  Speed 

150  kt 

153  kt 

Figure  7.  AH-64A  Specifications 

Helicopters,  as  well  as  most  other  types  of  equipment,  were  adversely  affected  by 
sand  and  other  environmental  conditions;  however,  methods  were  devised  to  control  the 
damage  and  to  maintain  a  high  rate  of  combat  readiness.  Operation  Desert  Stonn  was  the 
first  major  military  operation  conducted  on  a  largely  electronic  battlefield.  Army 
aviation  amply  demonstrated  its  effectiveness  in  this  environment  and  also  proved  again 
that  it  could  “own  the  night”  by  carrying  out  many  of  its  combat  operations  during 
darkness. 

The  reason  that  the  Apache  strike  force  team  needed  four  Air  Force  MH-53J  Pave 
Low  helicopters  to  help  start  the  Gulf  War  was  that  the  Apaches  needed  to  follow  the 
Pave  Lows  into  the  desert  due  to  the  Apache’s  lack  of  adequate  navigation  equipment 
capable  of  traversing  the  flat,  featureless  Mid  Eastern  terrain.  The  Apache  is  a  system 
that  continues  to  evolve;  even  today  there  are  deficiencies  and  shortcomings  that  are 
being  addressed. 
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Since  Desert  Storm,  Army  aviation  has  taken  part  in  several  other  operations: 
Provide  Comfort  in  northern  Iraq,  Restore/Continue  Hope  in  Somalia,  Uphold 
Democracy  in  Haiti,  and  the  NATO  operation  in  Bosnia.  Aviation  still  continues  to 
evolve,  and  Somalia  provided  important  lessons  learned  relating  to  military  operations  in 
an  urban  environment.  With  the  firepower  of  the  Apache,  Army  aviation  continues  to 
demonstrate  its  unique  worldwide  capability  through  a  combination  of  versatility, 
deployability,  and  lethality.  The  Apache’s  ability  to  evolve  allows  it  to  be  an  important 
ingredient  of  almost  any  type  of  contingency  operation  anywhere  in  the  world. 
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1970  AAH  work  begun 

January  -  August  1972:  Marks  Board  formed,  mission:  To  study  requirements  for  an 
attack  helicopter  (Chartered  to:  “Revalidate  the  Advanced  Aerial  Fire  Support  System 
Qualitative  Material  Requirement”) 

September  1972:  AAH  Material  Need  approved 

November  1972:  AAH  RFP  released 

February  1973:  RFP  responded  to  by  5  companies 

(Sikorsky,  Boeing- Vertol,  Bell  Helicopter,  Hughes,  and  Lockheed) 

April  1973:  AAH  PMO  stood-up  (BG  Samuel  G.  Cockerham,  1st  PM) 

June  1973:  Down  select  to  competitive  development  with  Hughes  and  Bell  Helicopter 

September  1975:  First  flight,  Bell’s  YAH-63A  &  Hughes’  YAH-64A 

April  1976:  New  AAH  PM  (MG  Edward  M.  Browne,  April  1976  -  December  1982) 

June  1976:  Prototypes  delivered  to  Army  for  flyoff 

December  10, 1976:  Down  select  to  Hughes  YAH-64A 

June  1981:  Operational  Test  (OT  II)  @  Hunter  Liggett  (Ft.  Ord,  CA) 

FY  1982:  Congress  approves  LRIP,  $444.5  M  Contract  for  1 1  aircraft 

November  1982:  Hughes  completes  $300  M  AAH  production  facility  in  Mesa,  AZ 

November  1982:  $106  Million  low  rate  production  contract  for  48  aircraft 

September  30, 1983:  First  production  aircraft  complete 

December  30, 1983:  Hughes  Helicopter  Company  sold  to  McDonnell  Douglas  Corp 
Spring  1984:  $841  Million  production  contract  for  112  aircraft 

Figure  8.  Apache  Development  Time  Line 
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III.  DATA 


A.  DATA  COLLECTION 

My  primary  research  question  is:  “What  were  the  key  aspects  of  the  Apache 
development  that  resulted  in  its  successful  employment  in  Desert  Storm?”  This  question 
will  be  addressed  by  looking  at  a  series  of  secondary  questions.  In  order  to  answer  these 
questions,  data  has  been  collected  in  a  structured  combination  of  “system”  information 
and  “technology”  infonnation.  To  facilitate  the  collection  of  data,  I  used  a  survey 
developed  by  a  research  team  lead  by  The  University  of  Alabama  in  Huntsville  Research 
Institute  and  Massachusetts  Institute  of  Technology  Sloan  School  of  Management  an 
example  of  a  completed  survey  can  be  found  in  Appendix  C. 

The  research  methodology  is  a  case  study.  As  such,  surveys  were  sent  to  a  very 
small  number  of  people  intimately  involved  with  the  development  and  production  of  the 
Apache  Helicopter.  The  study  must  be  done  now  because  many  of  the  men  and  women 
responsible  for  the  development  and  eventual  fielding  of  this  weapon  system  are  retiring, 
taking  with  them  important  knowledge  that  should  be  captured  and  codified  into  practical 
lessons  for  the  future.  The  data  has  been  collected  from  both  a  Government  and  a 
contractor  Program  Management  team  member.  Responses  from  survey  questions 
relevant  to  each  of  my  research  topics  are  organized  below.  This  chapter  is  non- 
traditional,  but  that  it  will  provide  the  foundation  for  my  analysis  and  a  combined  survey 
for  the  research  team. 

The  unique  numbering  system  from  the  survey  is  maintained  for  reference  and 
organizational  purposes.  The  questions  are  organized  by  topic,  the  major  phases  of  the 
program  are  defined  as  follows:  The  survey  is  centered  around  three  phases  in  a  major 
program.  First  is  the  systems  planning  and  pre-development  phase,  “SP”,  this  is  the  stage 
where  planning  work  began  on  the  integrated  system.  Systems  planning  ended  when 
contractors  were  selected  for  the  development  phase.  The  second  stage  is  the 
development  phase,  “D”,  in  this  phase  contractors  performed  advanced  engineering  and 
development.  A  fly-off  occurred  in  the  development  phase  with  the  winner  moving  on  to 
operational  test.  The  third  stage  is  referred  to  as  transition  to  production,  “TP”,  this  is 
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when  a  producible  system  prototype  has  been  demonstrated  in  an  operational 
environment  and  is  ready  for  low  rate  initial  production.  Questions  are  also  divided  into 
the  following  categories  as  defined  by  the  first  letter  in  the  question: 

•  “T”  stands  for  Technology 

•  “H”  stands  for  project  History 

•  “V”  stands  for  Validation  activities:  testing  and  simulation 

•  “D”  stands  for  team  participants  and  communications  during  Development 

•  “F”  stands  for  activity  report  during  system  development  stage  of  the 

project 

•  “W”  stands  for  When  activity  phasing  occurred  by  stages  of  development 

and  transition 

•  “B”  stands  for  problem  solving  and  team  effort 

•  “O”  stands  for  project  Outcomes 


The  definitions  of  the  various  organizations  of  interest  used  throughout  the 
document  are  defined  in  Table  1.  Scales  used  for  Technology  and  Production  Readiness 
are  defined  in  figures  9  and  10  respectively. 


Prime’s  S&T  org. 

The  Science  and  Technology  group  within  system  prime 

contractors  organization  responsible  for  doing  internal  research 

and  development  (IR&D)  and  developing  new  technology  and 

concepts 

Other  prime  org. 

Any  prime  contractor  organization  other  than  the  S&T 

organization 

Supplier  S&T 

Same  definition  as  for  prime’s  S&T  organization,  but  located  at  a 

supplier. 

Other  supplier  org. 

Any  supplier  organization  other  than  the  S&T  organization. 

Army  Lab/Center 

One  or  more  of  the  Army  laboratories  or  research,  development 

and  engineering  centers. 

Other  DoD/S&T  org. 

An  equivalent  of  an  Army  Lab/Center  found  elsewhere  in  DoD. 

Table  1.  Organization  Definitions 
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B.  RESEARCH  QUESTION  NUMBER  1:  USER  SUPPORT  AND  FUNDING 

“To  what  extent  was  there  user  support  and  funding  stability  during  system 
development?” 

This  question  will  be  addressed  by  looking  at  the  role  of  the  user,  requirements 
and  funding  stability.  Information  for  user  support  comes  from  survey  questions  D18, 
F5,  F6,  W3,  W4,  and  W5,  requirements  stability  from  F7,  W6  and  B13,  and  funding 
stability  from  questions  HI,  D1 1  and  B2. 

User  support,  survey  questions  D18,  F5,  F6,  W3  through  W5 

D18.  There  was  a  lot  of  contact  with  Training  &  Doctrine  Command  (TRADOC)  and/or 
other  appropriate  user  representatives  during  the  project? 

Government  Answer:  Strongly  Agree. 

Contractor  Answer:  Somewhat  Agree. 

How  often  did  the  following  occur  during  Development?  (F5  &  F6) 

F5.  Did  TRADOC/other  user  organizations  show  strong  support? 

Government  Answer:  Many  Times. 

Contractor  Answer:  Don’t  know. 

F6.  Were  there  changes  in  key  TRADOC  or  other  user  personnel? 

Government  Answer:  Several  Times. 

Contractor  Answer:  Don’t  know. 


ACTIVITY  PHASING  BY  STAGES  OF  DEVELOPMENT  AND  TRANSITION 
When  did  the  team  carry  out  the  following  activities? 


W3.  When  was  the  TRADOC  consulted  on  project  questions? 

Government  Answer:  Selection,  between  system  planning  phase  and 

development  phase. 

Contractor  Answer:  Later  Development,  between  development  phase  and 
transition  to  production  phase. 
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W4.  When  was  there  change  in  key  TRADOC/user  representatives? 

Government  Answer:  Early  Development,  between  development  and  transition  to 
production. 

Contractor  Answer:  Do  not  know. 

W5.  When  did  TRADOC/other  users  show  strong  support? 

Government  Answer:  Middle  Development,  between  development  and  transition 
to  production. 

Contractor  Answer:  Transition,  after  transition  to  production. 


Requirements  stability:  Survey  questions  F7,  W6  and  B13 

F7.  Were  there  changes  in  system  requirements  (e.g.,  threat)? 

Government  Answer:  Several  times. 

Contractor  Answer:  Do  not  know. 

W6.  When  was  there  change  in  the  system  requirements? 

Government  Answer:  After  development. 

Contractor  Answer:  Do  not  know. 

B13.  Did  this  problem  come  up  during  this  project?  Threat  definition  or  other 
requirements  changed  during  the  project. 

Government  Answer:  Significant  effort  spent  on  this  problem. 

Contractor  Answer:  Do  not  know. 

Funding  stability:  Survey  questions  HI,  Dll  and  B2 

HI.  At  some  point,  was  the  project  either:  1.  Slowed  down?  2.  Stopped  and  restarted? 
3.  Neither. 

Government  Answer:  1 .  Slowed  down,  due  to  funding  cuts. 

Contractor  Answer:  3.  Neither. 
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Dll.  There  was  often  uncertainty  about  the  future  of  project  funding  during  the  System 
Development  stage? 

Government  Answer:  Agree  somewhat,  the  funding  changes  were  a  constant 
concern  to  the  PMO,  this  caused  the  staff  to  be  concerned  at  each  major  milestone. 

Contractor  Answer:  Disagree  somewhat,  some  uncertainty  existed,  but  we  were 
confident  that  the  system  filled  a  need. 

B2.  Did  this  problem  come  up  during  this  project?  Cut-backs  in  project  resources  forced 
changes/compromises. 

Government  Answer:  Major  effort  spent  on  this  problem. 

Contractor  Answer:  Minor  effort  spent  on  this  problem. 

C.  RESEARCH  QUESTION  NUMBER  2:  CRITICAL  TECHNOLOGIES 

“To  what  extent  did  the  maturity  of  critical  technologies  being  integrated  into  the 
Apache  influence  development?  How  were  the  organizations  that  had  developed  the 
critical  technologies  involved  during  system  development?” 

This  question  will  be  addressed  by  looking  at  three  critical  technologies,  central  to 
the  success  of  the  developed  for  the  Apache,  in  terms  of  technology  readiness,  the  project 
timeline  and  project  outcomes.  Infonnation  for  critical  technologies  comes  from  survey 
question  Tl,  technology  readiness,  from  survey  questions  on  systems  planning  “S”, 
development  “D”,  transition  to  production  “TP”,  and  T5  through  T7,  and  project 
outcomes  from  survey  questions  01  through  09. 

The  second  question  will  be  addressed  by  looking  at  the  involvement  of 
development  organizations  throughout  system  development.  Survey  data  includes,  the 
role  of  the  S&T  organizations,  integration  difficulties,  production  readiness,  the  role  of 
the  prime  contractor,  and  problem  identification.  The  role  of  the  S&T  organizations 
comes  from  survey  questions  on  page  1  and  T8  through  T10.  Difficulties  integrating 
technologies  are  addressed  in  survey  questions  T3,  H3,  Bl,  B4  through  B8.  While 
production  readiness  issues  are  in  survey  questions  on  page  1,  T3,  H6,  B4  through  B6  and 
B8. 
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Identify  critical  technologies:  Survey  question  Tl. 

Now  identify  one  or  more  (up  to  3)  technologies  that  were  incorporated  into  the 
system  you  are  studying.  These  technologies  should  be  among  those  central  to  the 
success  of  the  system. 


Government  response: 

Technology  A 

Visionics 

Technology  B 

Computers 

Technology  C 

Avionics 

Contractor  response: 

Technology  A 

Target  Acquisition/Designation  (TADS) 

Technology  B 

Forward  Looking  Infrared  System  (FLIRS) 

Technology  C 

Hellfire  Missile  Integration  (Laser) 

Table  2.  Critical  Technologies  Response 

NOTE:  There  were  many  new  and  critical  technologies  incorporated  into  the  Apache.  In 
order  to  compare  answers,  participants  were  asked  to  respond  to  a  single  set  of  critical 
technologies.  Surveys  were  repeated  for  the  three  technologies  in  the  table  below. 


Technology  A 

Target  Acquisition/Designation  (TADS/PNVS) 

Technology  B 

Computers 

Technology  C 

Avionics 

Table  3.  Critical  Technologies  Evaluated 
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Technology  readiness:  Survey  questions  SP,  D,  TP,  and  T5  through  T7 

SP.  What  was  the  approximate  starting  date  of  systems  planning  and  pre-development 
work?  This  date  is  when  planning  work  began  on  the  integrated  system.  The  systems 
concept  and  applications  had  been  formulated,  but  applications  were  still  speculative. 
There  was  no  proof  or  detailed  analysis  to  support  the  approach. 


SP:  SYSTEMS  PLANNING  START  DATE  (SP): 

Government  Answer:  Cheyenne  Development  effort,  begun  in  1963,  preceded 
Apache  Development/Planning  work  begun  approximately  09/69  (mo/yr).  Much  of  this 
work  helped  jump-start  the  AAH  program. 

Contractor  Answer:  Approximately  1974. 


SP.  In  what  organization  was  the  primary  work  leading  up  to  this  point  accomplished? 
Government  Answer:  CHEYENNE  PM,  with  lead  support  from  TRADOC. 
Contractor  Answer:  Prime’s  S&T  organization. 


SP.  Including  that  organization,  what  organizations  had  been  involved  up  to  this  point? 

Government  Answer:  Active  support  from  both  other  prime  organizations  and 
other  supplier  organizations. 

Contractor  Answer:  Lead/co-lead  from  other  prime  organizations.  Active 
support  from  supplier  S&T  organizations,  the  Army  lab  centers,  and  other  DoD/S&T 
organizations. 


SP.  What  was  the  nature  of  the  Army  lab  center’s  involvement?  (Simulation?  Concept 
formulation?  Integration?  Requirements  development?) 

Government  Answer:  Most  of  the  behind  the  scene  “effort”  was  competing 
contractors  (Bell  and  Sikorsky),  supporting  their  King  Cobra  and  S-67  as  replacement  for 
the  LOCKHEED  CHEYENNE  while  supporting  tennination  of  the  CHEYENNE 
Production  contract  as  well  as  the  eventual  development  contract.  Definition  of  the 
Advanced  Attack  Helicopter  was  centered  around  what  the  Department  of  Army  thought 
Bell  Helicopter  was  capable  of  producing. 

Contractor  Answer:  Did  not  know. 


D.  Date  when  Development  started.  Typically  at  this  date,  funding  started  for  system 
advanced  development  or  engineering  development,  a  Government  project  office  was 
formed  and  prime  contractor(s)  selected. 

Government  Answer:  DEVELOPMENT  START  DATE:  (D):  June  1973. 

Contractor  Answer:  DEVELOPMENT  START  DATE:  (D):  1976. 
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D.  What  was  the  Technology  Readiness  Level  (refer  to  page  8)  for  the  SYSTEM  on  this 
date? 

Government  Answer:  6.  System/subsystem  model  or  prototypes  were 

demonstrated  in  a  relevant  environment.  Representative  model  or  prototype  systems 
were  tested,  which  is  well  beyond  the  breadboard  tested  for  TRL  5,  tested  in  a  relevant 
environment.  Represents  a  major  step  up  in  a  technology’s  demonstrated  readiness. 
Examples  include  testing  a  prototype  in  a  high  fidelity  laboratory  environment  or  in  a 
simulated  operational  environment. 

Contractor  Answer:  3.  The  system  is  in  the  analytical  and  experimental  critical 
function  and/or  characteristic  proof  of  concept  stage.  Analytical  and  laboratory  studies 
have  physically  validated  analytic  predictions  of  separate  elements  of  the  technology. 
Examples  include  components  that  are  not  yet  integrated  or  representative. 


D.  What  was  the  Production  Readiness  Level  (refer  to  page  8)  for  the  SYSTEM  on  this 
date? 

Government  Answer:  2.  The  application  is  produced  outside  the  lab  with  tools 
and  processes  used  for  producing  very  low  quantities. 

Contractor  Answer:  2.  The  application  is  produced  outside  the  lab  with  tools  and 
processes  used  for  producing  very  low  quantities. 


D.  In  what  organization  was  the  primary  work  in  the  period  from  systems  planning  to 
development  accomplished? 

Government  Answer:  Primary  work  was  accomplished  in  the  Project 

Management  office. 

Contractor  Answer:  Primary  work  was  accomplished  in  the  primes  science  and 
technology  organization. 


D.  Including  that  organization,  what  organizations  had  been  involved  in  the  period 
systems  planning  to  development? 

Government  Answer:  Involved  -  Other  prime  organizations.  Kept  informed  - 
Army  lab  center  and  other  DoD/S&T  organizations.  Not  Involved  -  Primes  S&T 
organization,  supplier  S&T,  and  “other  supplier  organizations. 

Contractor  Answer:  Lead/co-lead  -  Other  prime  organization  and  supplier  S&T 
organizations.  Active  support  from  the  Anny  lab  center” 
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D.  What  was  the  nature  of  the  Army  Lab/Center’s  involvement?  (Engineering  support? 
Simulation  or  testing?  Integration?  Requirements  interpretation?) 

Government  Answer:  Technical  consultation.  The  Night  Vision  Lab  was  the  lead 
support  for  the  TADS/PNVS. 

Contractor  Answer:  Engineering  support. 


TP.  Date  the  system  started  transition  to  production  “TP”.  This  is  when  the  producible 
system  prototype  has  been  demonstrated  in  an  operational  environment. 

Government  Answer:  TRANSITION  TO  PRODUCTION  (TP)  DATE:  Apr  1982. 

Contractor  Answer:  TRANSITION  TO  PRODUCTION  (TP)  DATE:  1980. 


TP.  What  was  the  Production  Readiness  Level  for  the  SYSTEM  on  this  date? 
Government  Answer:  3. 

Contractor  Answer:  4. 


TP.  In  what  organization  was  the  primary  work  in  the  period  from  development  to 
transition  to  production  accomplished? 

Government  Answer:  Program  Management. 

Contractor  Answer:  Prime  S&T  organization. 


TP.  Including  that  organization,  what  organizations  had  been  involved  in  the  period 
development  to  transition  to  production? 

Government  Answer:  Lead/co-lead  -  Prime  S&T  organization  and  other  prime 
organizations.  Active  support  -  Supplier  S&T  organization.  Involved  -  Other  supplier 
organizations,  the  Army  lab  center,  and  other  DoD/S&T  organizations. 

Contractor  Answer:  Lead/co-lead  -  Other  prime  organizations  and  supplier  S&T 
organizations.  Active  support  -  Army  lab  center. 


TP.  What  was  the  nature  of  the  Army  Lab  Center’s  involvement?  (Engineering  support? 
Simulation  or  testing?  Integration?  Requirements  interpretation?) 

Government  Answer:  Simulation. 

Contractor  Answer:  Engineering  Support;  Requirements  Definition. 


Now  look  at  the  SP,  D,  and  TP  dates  you  provided  above  for  the  System.  Using  the 
Technology  Readiness  (TRL)  Scale  (see  end  of  this  chapter),  find  the  number  that 
represents  the  readiness  of  the  separate  technologies  the  team  was  working  with  at  each 
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point  in  time.  Please  answer  here  for  the  state  of  development  of  each  component 
technology.  (NOT  for  the  over-all  system,  which  was  the  focus  in  the  questions  above.) 


Technology 

A 

TADS/PNVS 

Technology 

B 

Computers 

Technology 

C 

Avionics 

Government  response: 

T5.  When  System  planning  and  pre¬ 
development  began,  technology  TRL  was: 

2 

3 

7 

T6.  When  System  went  into  Development, 

technology  TRL  was: 

3 

3 

7 

T7.  When  System  reached  Transition  to 

Production,  technology  TRL  was: 

8 

9 

9 

Contractor  response: 

T5.  When  System  planning  and  pre¬ 
development  began,  technology  TRL  was: 

2 

2 

6 

T6.  When  System  went  into  Development, 

technology  TRL  was: 

4 

7 

7 

T7.  When  System  reached  Transition  to 

Production,  technology  TRL  was: 

8 

8 

8 

Table  4.  Technology  Readiness 


Project  Outcomes:  Survey  questions  Ol  through  09 

01.  Project  Acceptance.  Was  the  SYSTEM  accepted  to  be  put  into  production?  This  is 
initial  acceptance,  not  whether  it  actually  ended  up  in  production. 

Government  Answer:  3.  Yes,  the  System  was  accepted  for  production. 

Contractor  Answer:  3.  Yes,  the  System  was  accepted  for  production. 
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02.  After  the  SYSTEM  was  accepted  and  was  in  Transition  to  Production,  how  many 
additional  changes  in  the  designs  and  processes  were  later  required  before  the  System 
was  taken  into  full  production? 

Government  Answer:  2.  Significant  changes  were  made. 

Contractor  Answer:  3.  Minor  changes  were  made. 


03.  Did  the  SYSTEM  go  into  full  production? 

Government  Answer:  3.  Yes,  the  System  was  put  into  full  production. 
Contractor  Answer:  3.  Yes,  the  System  was  put  into  full  production. 


04.  For  each  of  the  technologies  A,  B,  and  C  above,  to  what  extent  was  each  used  in  the 
system  as  it  was  produced? 


Technology  A 

Technology  B 

Technology  C 

TADS/PNVS 

Computers 

Avionics 

Government  response: 

4.  Yes,  the  technology  was  used 

as  planned. 

X 

X 

X 

Contractor  response: 

4.  Yes,  the  technology  was  used 

as  planned. 

X 

X 

X 

Table  5.  Technology  Usage 


05.  After  the  SYSTEM  reached  Transition  to  Production,  did  the  project  go  to 
production  as  quickly  as  it  should  have? 

Government  Answer:  2.  One  to  six  months  delay 

Contractor  Answer:  2.  One  to  six  months  delay 
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06.  After  the  SYSTEM  was  actually  in  production,  how  many  additional  changes  in 
designs  and  processes  were  required? 

Government  Answer:  2.  Significant  changes. 

Contractor  Answer:  3.  Minor  changes. 

07.  Did  the  SYSTEM  as  it  was  implemented  meet  the  program’s  cost  goals? 

Government  Answer:  2.  The  results  came  close  to  achieving  cost  goals. 
Contractor  Answer:  2.  The  results  came  close  to  achieving  cost  goals. 

08.  Did  the  System  Development  program,  as  implemented,  come  in  on  budget? 
Government  Answer:  3.  The  project  significantly  exceeded  budget. 

Contractor  Answer:  9.  Don’t  know. 

09.  Did  the  System  as  it  was  implemented  meet  the  project’s  technical  goals  and 
functional  requirements? 

Government  Answer:  1 .  The  results  met  or  exceeded  technical  goals. 

Contractor  Answer:  1 .  The  results  met  or  exceeded  technical  goals. 


09x.  Did  the  System  have  problems  in  the  field  under  operational  conditions  in  Desert 
Storm?  IF  YOU  CHECKED  “1”  or  “2”,  what  did  the  field  problems  result  from?  Check 
all  that  apply. 

Government  Answer:  2.  Yes,  problems  in  the  field  caused  minor  problems  in  the 
system’s  effectiveness.  9b.  Requirements  did  not  reflect  the  field  environment. 

Contractor  Answer:  4.  No,  the  system  was  deployed  and  exceeded  expectations  of 
its  effectiveness. 


Role  of  the  S&T  organizations:  Survey  questions  page  1  (addressed  above), 
T8  through  T10,  and  Bll 
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For  each  of  the  technologies  A,  B  &  C,  did 

an  Army  Laboratory  or  Center  make  a 

significant  contribution  to  achieving  any 

of  the  above  levels  of  technology 

readiness? 

Technology 

A 

TADS 

/PNVS 

Technology 

B 

Computers 

Technology 

C 

Avionics 

Government  response: 

T9.  Yes,  it  contributed  to  Readiness  for 

Development. 

X 

X 

T10.  Yes,  it  contributed  to  Readiness  for 

Transition  to  Production. 

X 

Contractor  response: 

T8.  Yes,  it  contributed  to  Readiness  at  start 

of  Planning/Pre-development. 

X 

X 

X 

T9.  Yes,  it  contributed  to  Readiness  for 

Development. 

X 

X 

X 

T10.  Yes,  it  contributed  to  Readiness  for 

Transition  to  Production. 

X 

X 

X 

Table  6.  Army  Lab  or  Center  Involvement 


B1 1.  Army  Labs/Centers  resisted  project  ideas  or  approaches.  Did  this  problem  come  up 
during  this  project?  IF  YES,  how  much  project  effort  had  to  be  spent  on  this  problem? 

Government  Answer:  Yes,  minor  effort. 

Contractor  Answer:  No,  contractor  not  aware  of  any  problems. 


Difficulties  integrating  technologies:  Survey  questions  T3,  H3,  Bl,  B4 
through  B8 
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T3.  Production  Impact.  What  was 
the  impact  of  the  technology  on 
then  existing  production  processes? 
(Answer  for  date  you  provided  for 
Development  start,  D.) 

Technology  A 

TADS  /PNVS 

Technology  B 

Computers 

Technology  C 

Avionics 

Government  response: 

Technology  forced  deep  and  serious 

production  process  change. 

X 

X 

X 

Contractor  response: 

Technology  caused  significant 

production  process  change. 

X 

X 

Technology  did  not  require  much 

production  process  change 

X 

Table  7.  Production  Impact  of  Critical  Technology 

H3.  Key  Skills.  This  question  asks  about  “key  skills”  essential  to  the  success  of  the 
project,  defined  as  skills  “that  if  they  were  not  available  at  all,  would  have  stopped  team 
progress  at  the  point  when  they  were  needed.”  Were  there  any  key  skills  not  adequately 
represented  on  the  team?  IF  YES,  what  were  the  missing  key  skills?  Please  check  (  S  ) 
any  and  all  that  apply. 

Government  Answer:  Yes,  more  than  one.  The  team  lacked  producibility 
professionals  from  suppliers  and  Government  logistics  experts. 

Contractor  Answer:  No. 


For  the  following  questions,  did  this  problem  come  up  during  this  project?  IF  YES,  how 
much  project  effort  had  to  be  spent  on  this  problem? 

B 1 .  It  was  harder  than  expected  to  take  the  risk  out  of  the  new  technology. 

Government  Answer:  No,  much  to  the  risk  areas  had  been  identified. 

Contractor  Answer:  Minor  effort. 

B4.  A  critical  production  issue  was  uncovered  very  late  in  the  process. 
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Government  Answer:  Significant  effort. 

Contractor  Answer:  Minor  effort. 

B5.  Management  pressure  pushed  technology  prematurely  into  production. 

Government  Answer:  Major  effort. 

Contractor  Answer:  No. 

B6.  There  was  a  lack  of  acceptance  standards  for  the  new  technology. 

Government  Answer:  Minor  effort. 

Contractor  Answer:  No. 

B7.  The  technology  was  hard  to  scale  up  from  lab  and  pilot  tests. 

Government  Answer:  Minor  effort. 

Contractor  Answer:  Minor  effort. 

B8.  Testing,  quality  control  and/or  acceptance  took  longer  than  planned. 

Government  Answer:  Significant  effort. 

Contractor  Answer:  Minor  effort. 

Production  Readiness:  Survey  questions  Pagel  (addressed  above),  T3,  H6,  B4 
through  B6  and  B8 


T3.  Production  Impact.  What  was  the  impact  of  the  technology  on  then  existing 
production  processes?  (These  results  are  in  Table  6  above.) 

H6.  Whose  facilities  were  going  to  be  the  primary  production  site  for  the  application  of 
the  new  technologies? 

Government  Answer:  3.  Supplier  facilities. 

Contractor  Answer:  2.  Both  prime  and  supplier  facilities. 


For  the  following  questions,  did  this  problem  come  up  during  this  project?  IF  Yes,  how 
much  project  effort  had  to  be  spent  on  this  problem? 
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B4.  A  critical  production  issue  was  uncovered  very  late  in  the  process. 

Government  Answer:  Y es,  significant  effort  had  to  be  spent  on  this  problem. 
Contractor  Answer:  Y es,  minor  effort  had  to  be  spent  on  this  problem. 

B5.  Management  pressure  pushed  technology  prematurely  into  production. 

Government  Answer:  Yes,  major  effort  had  to  be  spent  on  this  problem. 
Contractor  Answer:  No. 

B6.  There  was  a  lack  of  acceptance  standards  for  the  new  technology. 

Government  Answer:  Yes,  minor  effort  had  to  be  spent  on  this  problem. 
Contractor  Answer:  No. 

B8.  Testing,  quality  control  and/or  acceptance  took  longer  than  planned. 

Government  Answer:  Y es,  significant  effort  had  to  be  spent  on  this  problem. 
Contractor  Answer:  Yes,  minor  effort  had  to  be  spent  on  this  problem. 

D.  RESEARCH  QUESTION  NUMBER  3:  TEST  STRATEGY 

“Did  the  test  strategy  adequately  evaluate  the  system  for  operational  use?”  This 
question  will  be  addressed  by  looking  at  the  test  approach  utilized  supplemented  with 
data  from  program  files  and  test  reports.  The  test  approach  is  covered  in  survey  questions 
VI  through  VI 5. 

Identify  test  approach:  Survey  questions  VI  through  V15 

VI.  Was  a  failure  modes  and  effects  analysis  done  on  the  system?  If  yes,  was  it  used  to 
help  establish  the  test  plan? 

Government  Answer:  Yes.  Yes,  it  was  used  to  help  establish  the  test  plan. 
Contractor  Answer:  Yes.  Don’t  know. 

These  questions  are  for  individual  components: 
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V2.  Was  there  testing  to  see  if  the  individual  components  of  the  system  worked?  What 
organization(s)  did  this  testing? 

Government  Answer:  Yes.  Prime  and  Suppliers  did  this  testing. 

Contractor  Answer:  Yes.  Prime,  Suppliers  and  Army  center/lab  did  this  testing. 

V3.  Were  there  simulations  run  to  see  if  the  individual  components  of  the  system 
worked?  What  organization(s)  did  these  simulations? 

Government  Answer:  Yes,  Prime,  suppliers  and  Army  center  lab  did  these 
simulations. 

Contractor  Answer:  Yes,  Prime  and  suppliers  did  these  simulations. 

For  integrated  components  in  controlled  setting: 

V4.  Were  the  components  tested  working  together  in  a  controlled  setting?  What 
organization(s)  did  this  testing? 

Government  Answer:  Yes.  Prime,  Suppliers  and  Army  center/lab  did  this  testing. 
Contractor  Answer:  Yes.  Prime  and  Suppliers  did  this  testing. 

V5.  Were  there  simulations  of  the  components  working  together  in  a  controlled  setting? 
What  organization(s)  did  this? 

Government  Answer:  Yes.  Prime,  Suppliers  and  to  limited  degree,  the  Army 
center  lab  performed  simulations. 

Contractor  Answer:  Yes.  Prime  and  Suppliers  performed  simulations. 


For  integrated  components  in  a  realistic  setting: 

V6.  Was  there  testing  of  the  components  working  together  in  a  realistic  setting?  What 
organization(s)  did  this  testing? 

Government  Answer:  Yes.  The  prime,  suppliers  and  the  Anny  lab  center 
performed  testing  in  a  realistic  environment. 

Contractor  Answer:  Yes.  The  prime  and  other  Government  organizations 

performed  testing  in  a  realistic  environment. 
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V7.  Was  a  hardware-in-the-loop  type  systems  integration  simulation  laboratory  used  to 
see  if  the  individual  components  of  the  system  worked?  To  see  if  integrated 
components  worked  in  controlled  setting? 

Government  Answer:  Yes  /  Yes. 

Contractor  Answer:  Yes  /  Yes . 

V8.  Recalling  the  total  effort  (100%)  spent  on  testing  and  simulations,  please  allocate  the 
percent  of  that  total  that  were: 

Gov  /  Cont  (Contractor  had  no  response) 

10  / _ %  spent  to  see  if  the  individual  components  of  the  system  worked. 

65  / _ %  spent  to  see  if  integrated  components  worked  in  controlled  setting. 

20  / _ %  spent  to  see  if  integrated  components  worked  in  a  realistic  setting. 

05  / _ %  spent  on  any  other  validation  purpose. 


Please  evaluate  the  following  statements  about  the  use  of  testing  and  simulations: 

V9.  Knowledge  from  validation  work  was  used  consistently  to  improve  components  and 
system. 

Government  Answer:  Neither  agree  nor  disagree. 

Contractor  Answer:  Agree  somewhat. 

V10.  Project  test  philosophy  was  to  “Break  it  big  early.” 

Government  Answer:  Strongly  disagree. 

Contractor  Answer:  Neither  agree  nor  disagree. 

V 1 1 .  Component  and  system  maturity  were  validated  at  the  right  times  in  the  program. 
Government  Answer:  Disagree  somewhat. 

Contractor  Answer:  Agree  somewhat. 

V12.  The  project  and  the  testing  community  had  an  adversarial  relationship. 

Government  Answer:  Agree  somewhat. 

Contractor  Answer:  Strongly  disagree. 
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V13.  Most  project  validation  events  produced  quality  results. 

Government  Answer:  Disagree  somewhat. 

Contractor  Answer:  Agree  somewhat. 

V14.  The  project  didn’t  recognize  important  lessons  that  validation  work  uncovered. 
Government  Answer:  Neither  agree  nor  disagree. 

Contractor  Answer:  Disagree  somewhat. 

V15.  Sometimes  the  project  settled  for  less  than  the  best  validation  method. 

Government  Answer:  Agree  somewhat. 

Contractor  Answer:  Disagree  somewhat. 

03.  Did  the  SYSTEM  go  into  full  production? 

Government  Answer:  3.  Yes,  the  System  was  put  into  full  production. 

Contractor  Answer:  3.  Yes,  the  System  was  put  into  full  production. 

05.  After  the  SYSTEM  reached  Transition  to  Production,  did  the  project  go  to 
production  as  quickly  as  it  should  have? 

Government  Answer:  2.  One  to  six  months  delay 

Contractor  Answer:  2.  One  to  six  months  delay 

09x.  Did  the  System  have  problems  in  the  field  under  operational  conditions  in  Desert 
Storm?  IF  YOU  CHECKED  “1”  or  “2”,  what  did  the  field  problems  result  from?  Check 
all  that  apply. 

Government  Answer:  2.  Yes,  problems  in  the  field  caused  minor  problems  in  the 
system’s  effectiveness.  9b.  Requirements  did  not  reflect  the  field  environment. 

Contractor  Answer:  4.  No,  the  system  was  deployed  and  exceeded  expectations  of  its 
effectiveness. 

E.  RESEARCH  QUESTION  NUMBER  4:  TEAMS 

“How  effectively  were  teams  employed  during  development?”  This  question  will 
be  addressed  by  looking  at  the  effectiveness  of  teams,  proper  staffing,  and  relationships 
between  manufacturing  and  suppliers.  Teaming  approach  utilized  is  addressed  in  survey 
questions  H2,  H4,  H5,  Dl,  D7,  D9,  D13,  D14,  D16,  D19,  and  F4. 
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Teaming  approach  utilized:  Survey  questions  H2,  H4,  H5,  D7,  D9,  D13,  D14, 
D16,  D19,  and  F4 

H2.  Was  the  project  set  up  as  a  cross-functional  integrated  product  team  (IPT),  a  project 
team  drawn  from  different  parts  of  the  contractor’s  organization  with  most  of  the  skills 
needed  for  the  development? 

Government  Answer:  No.  However,  it  was  used  for  production. 

Contractor  Answer:  No.  We  used  smaller  concentrated  teams  with  PM  over  site. 

H4.  During  the  Development  stage  of  the  project,  how  many  people  on  the  team  were 
collocated  very  close  together?  (On  the  same  floor  of  a  building  within  a  one-minute 
walk  of  each  other.) 

Government  Answer:  None. 

Contractor  Answer:  Some,  over  a  third. 

H4a.  Including  the  above,  how  many  people  on  the  team  were  collocated  in  the  same 
building? 

Government  Answer:  Few  were  collocated  in  the  same  building. 

Contractor  Answer:  Most,  2/3rds  or  more  were  collocated  in  the  same  building. 
H5.  How  many  people  on  the  team  involved  in  the  development  stage  had  worked  before 
with  others  on  the  project? 

Government  Answer:  Most,  two  thirds  of  the  people  or  more  had  worked  before 
with  others  on  the  project. 

Contractor  Answer:  Some,  over  a  third  had  worked  before  with  others  on  the 

project. 


Team  Participants  &  Communications  during  Development. 

Here  are  some  statements  about  the  people  on  the  project  during  the  system  development 
stage.  Please  indicate  your  level  of  agreement  or  disagreement  that  each  statement  is  a 
description  of  team  processes  on  this  project. 

D7.  Team  meetings  were  sometimes  frustrating  and  non-productive. 

Government  Answer:  Strongly  agree. 

Contractor  Answer:  Neither  agree  or  disagree. 
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D9.  Project  results  did  not  take  advantage  of  the  team’s  best  ideas. 

Government  Answer:  Disagree  somewhat. 

Contractor  Answer:  Disagree  somewhat. 

D13.  Management  project  reviews  were  constructive  &  helpful. 

Government  Answer:  Neither  agree  or  disagree. 

Contractor  Answer:  Agree  somewhat. 

D 14.  Formal  reviews  were  conducted  at  key  decision  points. 

Government  Answer:  Strongly  agree. 

Contractor  Answer:  Agree  somewhat. 

D 16.  Usually  team  knew  right  away  where  to  get  necessary  outside  help. 

Government  Answer:  Agree  somewhat. 

Contractor  Answer:  Agree  somewhat. 

D19.  The  Government  PM  was  reluctant  to  share  problems  with  Army  leaders. 

Government  Answer:  Agree  somewhat. 

Contractor  Answer:  Disagree  somewhat. 


Activity  Report  during  System  Development  Stage  of  Project, 

How  often  did  team  members  do  the  following  during  Development? 

F4.  Needed  management  help  to  resolve  project  team  disagreements? 

Government  Answer:  Many  times. 

Contractor  Answer:  Several  times. 

Proper  Staffing:  Survey  questions  H3,  D1  -  D6,  D8,  DIO 
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H3.  Key  Skills.  Defined  as  skills  “that  if  they  were  not  available  at  all,  would  have 
stopped  team  progress  at  the  point  when  they  were  needed.”  Were  there  any  key 
skills  not  adequately  represented  on  the  team? 

Government  Answer:  Yes,  more  than  one.  The  team  lacked  producibility 
professionals  from  suppliers  and  Government  logistics  experts. 

Contractor  Answer:  No. 

D 1 .  The  team  leader  was  good  at  resolving  technical  disagreements. 

Government  Answer:  Strongly  agree. 

Contractor  Answer:  Neither  agree  or  disagree. 

D2.  The  team  leader  was  good  at  getting  necessary  resources. 

Government  Answer:  Strongly  agree. 

Contractor  Answer:  Agree  somewhat. 

D3.  There  was  a  lot  of  turnover  in  team  membership. 

Government  Answer:  Strongly  disagree. 

Contractor  Answer:  Neither  agree  or  disagree. 

D4.  The  team  leader  had  both  design  &  production  experience. 

Government  Answer:  Strongly  disagree. 

Contractor  Answer:  Neither  agree  or  disagree. 

D5.  The  team  leader  had  very  high  technical  competence. 

Government  Answer:  Agree  somewhat. 

Contractor  Answer:  Agree  somewhat. 

D6.  Some  key  technical  skills  were  not  represented  on  the  team  itself. 

Government  Answer:  Strongly  disagree. 

Contractor  Answer:  Disagree  somewhat. 

D8.  Professionals  were  split  across  too  many  different  tasks  &  teams. 

Government  Answer:  Neither  agree  or  disagree. 

Contractor  Answer:  Neither  agree  or  disagree. 
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DIO.  Key  members  continued  through  pre-production  planning  and  testing. 

Government  Answer:  Strongly  agree. 

Contractor  Answer:  Agree  somewhat. 

Relationships  with  manufacturing  and  suppliers:  Survey  questions  F1-F3, 
F10-F13,  W1-W2,  W16-W18 

Activity  Report  during  System  Development  Stage  of  Project, 

How  often  did  team  members  do  the  following  during  Development? 

F 1 .  Went  to  the  shop  floor  to  meet  about  related  production  processes. 

Government  Answer:  Many  times. 

Contractor  Answer:  Several  times. 

F2.  Asked  for  supplier  comments  &  suggestions  on  design  choices. 

Government  Answer:  Many  times. 

Contractor  Answer:  Several  times. 

F3.  Showed  and  discussed  physical  models  of  new  components  with  suppliers. 
Government  Answer:  Many  times. 

Contractor  Answer:  Several  times. 


SHARED  DESIGN-PRODUCTION  ACTIVITIES  during  System  Development. 
Here  only  count  joint  meetings  or  discussions  that  included  both  DESIGN  and  people 
from  PRODUCTION  and/or  from  the  PROGRAM  concerned  with  production  of  the 
System.  How  often  did  team  members  do  the  following  during  Development? 

F10.  Passed  around  physical  prototypes  during  joint  discussions. 

Government  Answer:  Several  times. 

Contractor  Answer:  Several  times. 
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FI  1 .  Held  planning  meetings  that  included  both  design  &  production  people. 
Government  Answer:  Several  times. 

Contractor  Answer:  Several  times. 

F12.  Explored  choices  together  with  computational  models  or  analytic  tools. 
Government  Answer:  Several  times. 

Contractor  Answer:  Several  times. 

F13.  Had  test  articles  or  pre-production  parts  to  discuss  and  examine  jointly. 
Government  Answer:  Several  times. 

Contractor  Answer:  Several  times. 

ACTIVITY  PHASING  BY  STAGES  OF  DEVELOPMENT  AND  TRANSITION. 

When  did  the  team  carry  out  the  following  activities? 

W 1 .  When  did  production  representatives  participate  regularly? 

Government  Answer:  Later  in  development. 

Contractor  Answer:  Later  in  development. 

W2.  When  did  team  members  meet  with  production  on  shop  floor? 

Government  Answer:  During  Transition  to  Production. 

Contractor  Answer:  Later  in  development. 

Relationship  &  Activities  between  Engineering  Design  &  Production/Program 

These  questions  are  different  because  they  focus  only  on  joint  meetings  or  discussions 
that  included  both  DESIGN  personnel  and  people  from  PRODUCTION  and/or 
PROGRAM  people  concerned  with  production. 
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W16.  When  did  the  team  &  technical  professionals  from  Production  have  unscheduled 
&  informal  joint  conversations  about  the  project? 

Government  Answer:  During  Transition  to  Production. 

Contractor  Answer:  Later  in  development. 

W17.  When  were  analytic  engineering  tools  used  jointly  by  design? 

Government  Answer:  During  Transition  to  Production. 

Contractor  Answer:  Later  in  development. 

W18.  When  were  prototypes  and  parts  used  in  joint  discussions? 

Government  Answer:  During  Transition  to  Production. 

Contractor  Answer:  Later  in  development. 

F.  RESEARCH  QUESTION  NUMBER  5:  KEY  PM  ISSUE 

What  was  the  key  issue  that  the  PM  had  to  deal  with  during  the  project  and  how 
was  it  dealt  with?  This  question  will  be  addressed  by  looking  at  the  primary  issue  that 
the  PM  focused  most  attention  and  resources.  The  key  PM  issue  is  addressed  in  survey 
question  12. 


Key  PM  Issue:  Survey  question  12 

Government  Answer:  The  PM  controlled/dictated  the  R&D  Program.  Had  he  not 
dictated  the  R&D  program  there  would  not  be  an  AAH  today.  That  does  not  make  it 
correct  or  acceptable.  Control  of  the  production  project  was,  in  my  estimation,  the 
biggest  fundamental  problem  the  PM  had  to  deal  with  in  managing  the  overall  program. 
Problem  of  control  was  basically  the  external  environment,  i.e.,  the  sheer  number  of 
agencies,  that  were  to  be  contended  with  on  a  regular  basis  under  the  “team”  approach. 
Like  the  internal  organization,  each  had  their  own  axe  to  grind  and  each  had  some  level 
of  input.  As  an  example,  meetings  were  inordinately  large  and  therefore  more  difficult  to 
control.  Decisions  that  should  have  been  made  instantly  were  negotiated  to  death  leaving 
cost  and  schedule  impacts  to  be  resolved.  Each  PM’s  style  was  significantly  different 
and  varied.  During  R&D  the  military  were  given  no  more  than  lip  service.  During 
initial/early  production  the  military  were  the  only  ones  to  input  to  the  PM.  The  period 
1990-1996  were  years  of  the  very  best  management  in  the  program.  The  military  were 
competent  leaders  and  did  pay  close  consideration  to  the  civilian  technical  support. 

Contractor  Answer:  Don’t  know. 
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1.  Basic  principles  observed  and  reported.  Scientific  research  begins  to  be  translated 
into  applied  research  and  development  concepts.  There  have  been  paper  studies  of 
technology’s  basic  properties. 

2.  Technology  concept  and/or  application  formulated.  Practical  applications  have 
been  invented.  Application  is  speculative  and  there  is  no  proof  or  detailed  analysis  to 
support  the  assumptions.  Examples  are  still  limited  to  paper  studies. 

3.  Analytical  &  experimental  critical  function  and/or  characteristic  proof  of 
concept.  Analytical  and  laboratory  studies  have  physically  validated  analytic 
predictions  of  separate  elements  of  the  technology.  Examples  include  components 
that  are  not  yet  integrated  or  representative 

4.  Component  and/or  bread  board  validation  in  lab  environment.  Basic 
technological  components  are  integrated  to  establish  that  pieces  will  work  together, 
e.g.,  integration  of  ad  hoc  parts  in  lab.  This  is  relatively  “low  fidelity”  compared  to 
the  eventual  system. 

5.  Components  and/or  bread  board  validation  in  relevant  environment.  Fidelity  of 
breadboard  technology  is  significantly  increased.  Basic  components  integrated  with 
reasonably  realistic  supporting  elements  so  the  technology  can  be  tested  in  a 
simulated  environment.  Examples  include  “high  fidelity”  laboratory  integration  of 
components. 

6.  System/subsystem  model  or  prototype  demonstrated  in  a  relevant  environment. 

Representative  model  or  prototype  system,  which  is  well  beyond  the  breadboard 
tested  for  TRL  5,  tested  in  a  relevant  environment.  Represents  a  major  step  up  in  a 
technology’s  demonstrated  readiness.  Examples  include  testing  a  prototype  in  a  high 
fidelity  laboratory  environment  or  in  a  simulated  operational  environment. 

7.  System  prototype  demonstrated  in  an  operational  environment.  Prototype  near 
or  at  planned  operational  system.  Represents  a  major  step  up  from  TRL  6,  requiring 
the  demonstration  of  an  actual  system  prototype  in  an  operational  environment,  such 
as  in  an  aircraft,  vehicle  or  space.  Examples  include  testing  the  prototype  in  a  test  bed 
aircraft. 

8.  Actual  system  completed  and  qualified  in  test  and  demonstration.  Technology 
proven  to  work  in  final  form  and  under  expected  conditions.  In  almost  all  cases,  this 
TRL  represents  the  end  of  true  system  development.  Examples  include  developmental 
test  and  evaluation  in  its  intended  weapon  system  to  determine  if  it  meets  design 
specification. 

9.  Actual  system  proven  in  successful  operational  environment.  Actual  application 
of  the  technology  in  its  final  fonn  and  under  mission  conditions,  such  as  those 
encountered  in  operational  test  and  evaluation.  In  almost  all  cases,  this  is  the  end  of 
the  last  “bug  fixing”  aspects  of  true  system  development.  Examples  include  using  the 
system  under  operational  mission  conditions. 


Figure  9.  Technology  Readiness  Level  Scale 
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1.  The  subsystem  or  component  application  embodying  the  technology  is  produced 
inside  the  lab  by  engineers,  scientists  or  laboratory  technicians  to  demonstrate 
principles  for  breadboard  validation  and  testing. 

2.  The  application  is  produced  outside  the  lab  with  tools  and  processes  used  for 
producing  very  low  quantities. 

3.  The  application  is  produced  in  low  quantities  with  tools  and  processes  planned  to  be 
used  in  production  systems.  Testing  procedures  for  components  and  subsystems  are 
established. 

4.  The  system  involving  the  technology  application(s)  is  engineered  for  production.  All 
components  are  identified,  integration,  assembly  and  test  planning  is  complete. 

5.  Low  rate  production  has  been  run  using  the  production  processes  planned  for  full  rate 
production,  complete  with  validated  procedures  for  integration,  assembly  and  test  of  the 
system. 

Figure  10.  Production  Readiness  Scale 
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IV.  ANALYSIS 


A.  INTRODUCTION 

My  analysis  is  divided  into  two  parts,  a  combined  survey  contained  in  Appendix 
C  that  will  be  used  by  the  University  of  Alabama  Huntsville  /  Massachusetts  Institute  of 
Technology  researchers  for  a  crosscutting  analysis,  comparing  several  systems  used  in 
Desert  Storm,  and  the  written  analysis  contained  in  this  chapter. 

The  first  step  in  my  analysis  is  to  build  a  composite  questionnaire  of  the 
Government  and  contractor  responses.  To  complete  this  task,  I  first  looked  at  all  of  the 
questions  that  had  the  same  response.  If  this  answer  corresponded  to  my  research,  I 
accepted  it.  Where  responses  differed  from  my  research,  I  considered  the  sources  and 
made  a  logical  decision  as  to  which  was  the  best  answer.  This  mainly  occurred  with 
actual  dates.  To  resolve  conflicts  among  responses  I  looked  at  several  factors.  My  first 
judgment  was  based  on  the  resources;  the  reference  documents  that  I  used  are  contained 
in  Appendix  A.  If  no  additional  data  was  available,  I  considered  the  position  of  the 
respondents  with  respect  to  the  question  at  hand.  For  example,  when  discussing  user 
support,  I  relied  more  on  the  Government  response,  as  they  most  likely  had  much  more 
contact  with  the  user  community.  On  the  other  hand,  I  was  more  apt  to  use  the 
contractors’  response  for  questions  dealing  with  supplier  issues.  Where  more  information 
was  required,  I  conducted  follow-up  interviews  with  either  the  survey  respondents  or 
other  employees  familiar  with  the  project  during  the  specific  time  frame  in  question. 

This  chapter  provides  an  in-depth  analysis  of  the  data  obtained  from  the  survey 
questions  supplemented  with  other  research  data  as  necessary  to  clarify  the  results.  All 
data  was  organized  by  topic  in  the  previous  chapter  -  this  method  of  organization  will  be 
carried  forth,  survey  question  numbers,  [#],  are  annotated  throughout  so  the  reader  can 
reference  the  combined  questionnaire  in  Appendix  C,  research  citation  references  are 
listed  in  Appendix  A  {ref  #}.  Historical  events  from  Chapter  II  are  used  to  keep  the 
information  in  context  of  the  timeframe  wherein  events  occurred. 
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B. 


RESEARCH  QUESTION  NUMBER  1:  USER  SUPPORT  AND  FUNDING 


“To  what  extent  was  there  user  support  and  funding  stability  during  system 
development?”  This  question  will  be  addressed  by  looking  at  the  role  of  the  user, 
requirements  and  funding  stability.  I  will  primarily  focus  on  the  answers  from  the 
Government  side  of  the  Program  Management  Office  (PMO)  as  they  had  direct  contact 
with  the  Anny  user  community. 

The  PM  staff  had  a  good  working  relationship  with  the  users  and  frequently 
consulted  them  on  key  project  issues  [D18,  F5,  &  W3].  The  user  organizations  showed 
strong  support  during  Apache  development  [W5].  As  discussed  in  Chapter  2,  a  sense  of 
urgency  came  over  the  Army  leadership  following  the  rise  of  the  Air  Force  A- 10  program 
and  the  demise  of  the  Army’s  Cheyenne  program.  Survey  respondents  concurred  that 
combat  development  representatives  of  the  troops  in  the  field  were  on  board  early  in  the 
program  and  supported  the  program  as  it  evolved  [W3,  F5  &  W5].  Once  the  Apache  PM 
office  was  established  in  April  1973  [D],  the  PM  kept  in  close  contact  with  Training  and 
Doctrine  Command  (TRADOC)  and  user  representatives  during  the  remaining  phases  of 
the  project  [D18],  That  close  working  relationship  helped  user  support  to  grow  as  the 
program  progressed  toward  production.  As  is  usual  with  the  military  rotation  cycle,  there 
were  several  changes  in  key  user  personnel  during  the  program  [F6  &  W4].  Top 
leadership  helped  made  sure  that  these  changes  occurred  early  in  development  and 
between  development  and  the  transition  to  production  [F6  &  W4].  Keeping  key  user 
personnel  on  board  through  major  milestones  helped  to  minimize  the  effect  of  these 
inevitable  changes. 

Initially,  clear  requirements  helped  to  keep  the  program  on  course  [F7,  W6  & 
B 13].  The  Advanced  Attack  Helicopter  (AAH)  Mission  Needs  Statement  (MNS)  and 
operational  requirements  were  the  result  of  a  revalidation  of  the  Advanced  Aerial  Fire 
Support  System  qualitative  material  requirements  that  spawned  the  Cheyenne  program. 
The  new  mission  needs  statement  (MNS)  stipulated  that  the  AAH  would  be  in  production 
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by  1978.  {ref  2}  This  put  the  program  on  a  tight  schedule  from  the  start.  There  were 
several  new  technologies  on  the  horizon  that  could  not  be  integrated  in  time  to  meet  the 
fielding  date;  the  Hellfire  missile  contained  such  technology  [T 1  ].  The  PM  worked 
closely  with  the  user  community  to  build  a  program  that  would  meet  their  needs  [W3  & 
D18]  (e.g.,  being  able  to  fight  a  cold  war  battle  in  all  weather  conditions)  and  still  meet 
the  first  unit  equipped  (FUE)  timeline. 

It  was  said  that  the  PM  ruled  with  an  iron  fist  as  the  system  progressed  through 
development.  This  caused  great  consternation  throughout  the  technical  community,  but 
kept  the  program  on  course  [12].  Significant  effort  was  spent  on  controlling  the  problem 
of  requirements  creep  [F7,  W6  &  B13],  Although  there  were  changes  in  system 
requirements  [F7]  as  the  program  evolved,  such  as  the  laser  guided  Hellfire  missile  added 
in  February  1976  {ref  2},  the  close  working  relationship  between  the  user  and  PM  office 
[F5]  helped  foster  a  mutual  trust.  Most  significant  requirement  changes  were  kept  to  a 
minimum  as  the  program  progressed  through  development  and  on  into  production  [W6]. 
Many  requirement  changes  were  addressed  as  preplanned  product  improvements  after  the 
system  transitioned  to  production  {ref  3}. 

The  project  significantly  exceeded  initial  budget  estimates  [08].  Prior  to  approval 
of  a  large  program  by  the  Defense  Acquisition  Board  (DAB),  the  Office  of  the  Secretary 
of  Defense  has  their  Cost  Analysis  Improvement  Group  (CAIG)  provide  a  per  unit  cost 
estimate.  The  CAIG  is  chartered  to  provide  an  independent  review  of  life-cycle  cost 
estimates  and  to  determine  if  additional  analysis  is  required.  The  CAIG’s  flyaway  cost 
estimate  for  the  Apache  was  $1.7  million  in  fiscal  1972  dollars,  {ref  2} 

The  program  was  slowed  down,  due  to  funding  cuts  [HI].  As  the  Carter 
administration  took  the  reins  of  Government  in  Washington  in  1976,  the  Apache  flyaway 
cost  had  significantly  increased.  The  new  Secretary  of  Defense  in  the  Carter 
Administration,  Harold  Brown,  formerly  the  Secretary  of  the  Air  Force,  was  specifically 
the  one  who  pushed  for  the  A- 10  that  helped  kill  the  Cheyenne  program.  Brown  cut  the 
Apache  budget  by  one  half  on  the  second  week  of  the  Carter  Presidency,  {ref  2}  To 
make  up  for  the  funding  shortfall,  the  development  program  was  stretched  [HI,  Dl,  & 
B2]  an  additional  ten  months  {ref  1 } . 
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There  was  often  uncertainty  about  the  future  of  project  funding  during  the  system 
development  stage  [Dll].  Flyaway  costs  hit  $6.4  million  in  FY  1977  dollars  {ref  1}. 
The  1978  DoD  appropriations  bill  contained  only  half  the  requested  funding  for  the 
Apache;  the  program  was  almost  cancelled  {ref  2}.  Despite  the  cuts  the  program  pressed 
on  and  the  platform  proved  itself  during  subsequent  user  testing.  The  high  marks  that  the 
Apache  received  from  the  test  community  helped  greatly  when  the  program  moved  for 
production  approval.  In  1982,  Congress  authorized  $444.5  million  for  low  rate  initial 
production,  {ref  1} 

C.  RESEARCH  QUESTION  NUMBER  2:  CRITICAL  TECHNOLOGIES 

“To  what  extent  did  the  maturity  of  critical  technologies  being  integrated  into  the 
Apache  influence  development?  How  were  the  organizations  that  had  developed  the 
critical  technologies  involved  during  system  development?” 

This  question  will  be  addressed  by  looking  at  three  critical  technologies  central  to 
the  success  of  the  development  for  the  Apache  in  terms  of  technology  readiness,  the 
project  timeline  and  project  outcomes.  I  will  also  evaluate  the  involvement  of  various 
organizations  during  system  development.  Survey  data  includes:  the  role  of  the  various 
science  and  technology  (S&T)  organizations,  integration  difficulties,  production 
readiness,  the  role  of  the  prime  contractor  and  problem  identification. 

AAH  systems  planning  and  pre-development  work  started  in  1970,  soon  after  the 
Cheyenne  contract  was  terminated.  Government  respondents  stated  that  they  never  even 
changes  offices,  they  went  from  working  Cheyenne  one  day,  to  the  AAH  program  the 
next.  As  documented  in  chapter  2,  the  Cheyenne  had  tried  to  push  too  far  beyond  the 
current  state-of-art  technology;  this  caused  serious  cost  over  runs  and  schedule  slips.  The 
new  AAH  program  quickly  distanced  itself  from  the  Cheyenne,  {ref  2}  Technology 
readiness  was  a  key  factor  in  determining  the  capabilities  needed  for  the  new  aircraft 
[SP]. 

The  Apache  program  wanted  to  integrate  several  new  technologies  onto  its 
platform  [T 1  ] .  These  were  highly  sophisticated  subsystems  from  as  many  as  eight 
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different  PMO’s  along  with  a  number  of  subsystems  developed  by  the  prime  contractor 
and  other  suppliers  {ref  2}.  Initial  surveys  identified  several  of  these  different 
technologies,  all  in  the  area  of  mission  equipment  [T 1  ] .  Government  interviewees,  in 
response  to  the  survey,  identified  the  following  technologies:  visionics,  computers  and 
avionics  [T 1] .  Interviewees  from  the  contractor  community  responded  with  the  Target 
Acquisition/Designation  (TADS),  Forward  Looking  Infrared  System  (FLIRS),  and  the 
Hellfire  Missile  Integration  [T 1].  In  order  to  have  a  clear  view  of  technology  integration, 
I  will  concentrate  my  analysis  on  just  three  of  these  critical  technologies.  In  order  to 
have  consistent  answers,  participants  were  asked  to  respond  to  this  single  set  of  critical 
technologies  [T 1  ]. 

These  technologies  are  considered  to  be  among  those  central  to  the  success  of  the 
Apache  system,  that  is,  the  program  may  have  failed  if  these  technologies  were  not 
available  for  production.  They  are  as  follows:  the  Target  Acquisition  Designation 
System/Pilot  Night  Vision  System  (TADS/PNVS)  [T 1  ],  used  to  acquire  targets  in  all 
battlefield  conditions.  Second  are  the  computer  systems  [T 1  ] .  These  are  the  processors 
used  to  control  the  flow  of  information  on  the  platform.  The  third  critical  technology 
addressed  is  the  avionics  used  to  control  aircraft  flight  [T 1  ] . 

The  systems  planning  stage  of  the  program  started  with  a  revalidation  of  the 

requirements.  The  Government  respondent  noted  that  the  definition  of  the  Advanced 

Attack  Helicopter  was  centered  around  what  the  Department  of  the  Anny  thought  the 

helicopter  industry  was  capable  of  producing  [SP],  Great  care  was  taken  to  assure  that 

the  technologies  were  feasible  prior  to  sending  requests  for  proposals  to  industry.  The 

system  technology  readiness  level  (TRL)  was  at  level  2  [SP].  A  complete  definition  of 

the  nine  TRLs  can  be  found  in  figure  9,  page  44.  At  level  2  the  technology  concept  and 

application  had  been  fonnulated.  Practical  applications  had  been  invented.  However,  the 

application  was  speculative  and  there  was  no  proof  or  detailed  analysis  to  support  the 

assumptions.  Examples  were  still  limited  to  paper  studies.  At  the  subsystem  level,  the 

TADS/PNVS  and  computer  systems  needed  the  most  work  as  they  were  both  rated  at 

level  2  on  the  TRL  scale.  Some  systems  were  taken  off  the  critical  path  at  the  beginning 

of  development,  only  to  be  added  later  as  they  matured  {ref  2}.  The  avionics  system  was 

rated  a  level  6.  Avionics  system  and  subsystem  models  and  prototypes  were 
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demonstrated  in  a  relevant  environment.  Representative  models  and  prototype  systems, 
which  were  well  beyond  the  breadboard  tested  for  TRL  5,  had  been  tested  in  a  relevant 
environment.  This  represented  a  major  step  up  in  a  technology’s  demonstrated  readiness. 
Avionics  prototypes  had  been  tested  in  high  fidelity  laboratory  environments  and  in 
simulated  operational  environments,  that  is,  TRL  6  [D]. 

The  Government  respondent  confirmed  that  the  Army  labs  at  CECOM  and 
TRADOC  accomplished  the  primary  work  performed  in  the  period  from  system  planning 
to  development  with  oversight  from  the  Government  program  management  team 
remaining  from  Cheyenne.  Much  of  the  behind  the  scenes  effort  was  done  by  potentially 
competing  contractors  vying  for  the  replacement  of  the  Lockheed  Cheyenne  [SP].  Five 
companies  submitted  proposals  for  the  AAH.  They  were:  Sikorsky,  Boeing- Vertol,  Bell 
Helicopter,  Hughes,  and  Lockheed,  {ref  1} 

The  development  program  started  in  April  1973  [D],  as  the  AAH  PMO  was  first 
stood-up;  BG  Samuel  G.  Cockerham  was  the  first  PM  {ref  1}.  The  first  task  of  the  new 
PMO  was  to  down-select  the  proposals  received  towards  the  end  of  the  systems  planning 
stage.  Hughes  and  Bell  Helicopter  were  selected  for  competitive  development  {ref  1}. 
Each  would  build  prototypes  that  would  compete  in  a  “winner  take  all”  fly-off  for  the 
production  contract.  The  new  PMO  had  a  lot  of  work  to  do  in  a  very  short  period  of  time. 

At  the  start  of  development,  the  technology  readiness  level  (TRL)  for  the  overall 
system  was  estimated  to  be  at  level  5  [D],  that  is,  components  and/or  breadboard 
validation  had  been  done  in  a  relevant  environment.  Fidelity  of  breadboard  technology 
was  significantly  increased.  Basic  components  were  integrated  with  reasonably  realistic 
supporting  elements  so  the  technology  could  be  tested  in  a  simulated  environment. 
Examples  included  “high  fidelity”  laboratory  integration  of  components.  The  production 
readiness  level  (PRL)  for  the  system  at  this  point  was  at  best  level  2  [D].  A  complete 
definition  of  the  five  PRL’s  can  be  found  in  figure  10,  page  45.  Level  2  is  defined  as  the 
maturity  of  process  technology  sufficient  to  be  produced  outside  the  lab  with  tools  and 
processes  used  for  producing  very  low  quantities.  During  development,  the  Army  lab 
center’s  involvement  at  CECOM  evolved  around  engineering  support  and  requirements 
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interpretation  [D].  However,  in  the  case  of  the  TADS/PNVS,  the  Army’s  Night  Vision 
and  Electro-Optic  Lab  had  much  of  the  expertise  in  this  new  technology  [D]. 

Critical  technologies  were  in  several  different  levels  of  readiness  at  development 
start  [D],  Suppliers  were  quickly  trying  to  bring  their  subsystems  up  to  maturity  levels 
that  would  support  integration  into  other  systems  [D].  The  TADS/PNVS  and  computer 
systems  were  both  rated  at  level  4  on  the  TRL  scale  at  the  start  of  this  phase  [D].  Each 
had  been  through  component  and/or  breadboard  validation  in  a  lab  environment.  Basic 
technological  components  were  integrated  to  establish  that  pieces  would  work  together, 
e.g.,  integration  of  ad  hoc  parts  in  lab.  This  was  still  relatively  “low  fidelity”  compared 
to  the  eventual  system.  The  avionics  systems  were  much  more  advanced,  level  7,  as 
many  had  been  integrated  into  other  platforms  to  some  degree  [D].  Avionics  prototypes 
had  been  demonstrated  in  an  operational  environment.  These  prototypes  were  near  or  at 
a  planned  operational  state.  This  represented  a  major  step  up  from  level  6,  requiring  the 
demonstration  of  an  actual  system  prototype  in  an  operational  environment  in  an  aircraft. 
Avionics  prototypes  were  being  used  in  test  bed  aircraft  [D]. 

Apache  transition  to  production  occurred  in  April  1982  [TP].  This  was  defined  as 
the  period  when  the  producible  system  prototype  had  been  demonstrated  in  an  operational 
environment.  The  prototype  was  near  or  at  the  planned  operational  system  and  capable 
of  being  produced  on  a  small  scale.  Each  of  the  three  critical  technologies  had  also 
reached  technology  readiness  level  8  [TP].  The  Production  Readiness  Level  for  the 
system  on  this  date  had  grown  to  level  3,  as  it  was  produced  in  low  quantities  with  tools 
and  processes  that  had  been  planned  to  be  used  in  the  production  systems  [TP],  Testing 
procedures  for  components  and  subsystems  were  established,  while  subsystems  and 
support  had  not  been  finalized.  The  competing  prime  contractors’  science  and 
technology  organizations  accomplished  the  primary  work  in  the  period  from  development 
to  this  point.  Other  organizations  that  had  been  involved  in  the  period  included  active 
support  from  component  suppliers  and  the  CECOM  and  Night  Vision  Army  Lab  Centers 
[TP].  The  nature  of  the  Army  Lab  Center’s  involvement  was  engineering  support, 
simulation  and  testing  [TP].  Also  involved  were  other  supplier  organizations  while  other 
DoD  science  and  technology  organizations  were  kept  informed  of  the  progress  [TP]. 
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After  the  system  was  accepted  and  was  in  the  transition  to  production  phase, 
significant  changes  in  the  designs  and  processes  were  later  required  before  the  system 
was  taken  into  full  production  [TP].  Each  of  the  critical  technologies  was  used  as 
planned  in  the  final  system  [TP],  After  the  system  was  actually  in  production,  significant 
changes  in  designs  and  processes  were  also  required.  However,  the  results  of  the  system 
as  it  was  implemented,  met  or  exceeded  the  project’s  technical  goals  [TP], 

Survey  respondents  noted  that  the  system  experienced  some  problems  in  the  field 
under  operational  conditions  in  Desert  Storm  [TP].  Sand  and  dust  played  a  significant 
role  in  many  of  the  problems  {ref  3}.  The  Government  respondent  felt  that  these 
problems  were  caused  from  the  requirements  not  reflecting  the  true  field  environment 
[TP]. 

D.  RESEARCH  QUESTION  NUMBER  3:  TEST  STRATEGY 

“Did  the  test  strategy  adequately  evaluate  the  system  for  operational  use?”  This 
question  will  be  addressed  by  looking  at  the  test  approach  utilized. 

The  test  strategy  for  the  Apache  was  divided  into  several  phases  [SP,  D,  TP].  The 
initial  testing  for  Phase  I  of  the  program  involved  two  competing  contractor  designs.  As 
documented  in  chapter  2,  Hughes  Helicopter  and  Bell  Helicopters  were  each  awarded  a 
contract  in  June  of  1973  to  proceed  into  development.  The  designs  would  compete  in  a 
fly-off.  The  first  flight  for  both  aircraft  occurred  in  September  1975  followed  by  six 
months  of  contractor  testing.  Prototype  aircraft  were  delivered  to  the  Army  in  June  1976 
for  evaluation.  The  Hughes  design  won  the  competition  and  was  awarded  the  phase  II 
contract  in  December  1976.  {ref  1} 

The  Apache  program  entered  testing  with  the  failure  of  the  Cheyenne  program 
fresh  on  everyone’s  mind  {Ch  2}.  A  failure  modes  and  effects  analysis  was  done  on  the 
system.  This  analysis  was  performed  early  enough  for  the  results  to  be  used  to  establish 
the  test  plan.  The  failure  analysis  also  helped  establish  the  critical  test  parameters  for 
both  the  system  and  key  components.  [VI] 
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Several  organizations  were  involved  in  testing  the  various  components  that  were 
about  to  be  integrated  onto  the  Apache  [V2  -  V7].  Testing  and  simulations  were 
performed  first  to  see  if  the  individual  components  of  the  system  worked  [V3,  V5  &  V7]. 
The  prime  contractor,  component  suppliers  and  Army  labs  at  both  CECOM  (Avionics) 
and  the  Night  Vision  lab  (TADS/PNVS)  performed  the  bulk  of  this  testing  with  oversight 
from  the  PMO  [V2  -  V7], 

The  integrated  components  were  tested  working  together  in  a  controlled  setting 
[V4],  This  testing  takes  the  most  time,  as  problems  are  found,  fixed  and  retested.  To 
reduce  the  cost  of  retest,  simulations  were  also  perfonned  with  the  components  working 
together  in  a  controlled  setting  [V5].  The  prime  contractor,  suppliers  and  to  a  limited 
degree,  Anny  labs  performed  these  simulations  [V5].  A  hardware-in-the-loop  type 
systems  integration  simulation  laboratory  was  used  to  see  if  the  individual  components  of 
the  system  worked  and  to  see  if  integrated  components  worked  in  a  controlled  setting 
[V7]. 

As  the  system  evolved,  testing  was  perfonned  on  the  components  working 
together  in  a  realistic  setting  [V6].  The  organizations  that  perfonned  this  testing  included 
the  prime,  suppliers  and  Army  labs  [V6].  Once  all  the  bugs  were  worked  out,  the  system 
was  turned  over  to  the  Government  operational  testers  for  their  independent  evaluations. 
The  Apache  OT  II  was  perfonned  by  Army  pilots  and  occurred  from  June  to  August 
1981  at  Ft.  Hunter  Liggett,  {ref  8}  The  program  managers’  office  kept  a  constant  vigil 
over  the  testing.  To  accomplish  this,  the  Apache  PMO  established  a  field  office  at  the 
test  area,  {ref  1  &  8}  This  office  kept  the  PM  aware  of  what  was  going  on  at  the  test  site, 
quickly  resolved  problems  and  facilitated  the  flow  of  spare  parts  {ref  8}.  This 
relationship  helped  the  Apache  program  stay  on  course  and  get  through  operational 
testing  on  schedule  and  within  budget.  The  system  soon  advanced  to  Milestone  III  and 
approval  to  enter  into  production  [03  &  05]. 

There  were  several  environmental  issues  found  when  the  Apaches  were  first 
deployed  [09x].  When  they  fought  in  a  jungle,  water  intrusion  was  a  major  problem  {ref 
2}.  During  the  Gulf  War,  the  fine  sand  particles  caused  new  challenges  {ref  3}. 
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E.  RESEARCH  QUESTION  NUMBER  4:  TEAMS 

“How  effectively  were  teams  employed  during  development?”  This  question  will 
be  addressed  by  looking  at  the  use  of  teams,  proper  staffing,  and  interrelationships 
between  the  government,  manufacturing  and  suppliers. 

The  project  was  not  set  up  with  a  cross-functional  IPT,  that  is,  a  project  team 
drawn  from  different  parts  of  the  organization  with  most  of  the  skills  needed  for  the 
development  [H2].  The  current  trend  in  project  organization  is  to  form  cross-functional 
integrated  product  teams  (IPT).  This  is  used  to  assure  that  all  aspects  of  process 
integration  are  addressed.  The  Apache  program  development  occurred  during  the  1970’s, 
about  20  years  prior  to  the  use  of  the  formal  IPT  process  [D].  Instead  the  project  team 
had  smaller  technical  cells  each  concentrating  their  own  specific  piece  of  the  program. 
The  contractor’s  program  management  office  had  oversight  of  the  cells  and  was 
responsible  for  pulling  all  of  the  pieces  together  [H2]. 

According  to  the  survey  results,  nearly  two  thirds  of  the  people  on  the  contractor’s 
team  were  new  employees  and  thus,  had  never  even  worked  with  others  in  the  company 
until  the  Apache  development  [H5].  Attack  helicopter  development  was  new  to  Hughes 
Helicopters  {ref  2}.  During  the  development  stage  of  the  project,  the  contractor  had  just 
over  one  third  of  the  people  on  the  team  collocated  in  the  same  building  [H4a],  Few  were 
collocated  very  close  together,  that  is,  on  the  same  floor  of  a  building  within  a  one- 
minute  walk  [H4].  However,  key  technical  skills  were  well  represented  on  the  team  itself 
[H3].  Key  members  stayed  with  the  team  through  pre-production  planning  and  testing 
[DIO]. 

The  team  leaders  were  skillful  at  getting  necessary  resources  [D2].  Team  leaders 
were  fairly  effective  at  resolving  technical  disagreements  during  development  [D 1  ] . 
Turnover  in  team  membership  was  minimized  [D3  &  DIO].  Team  leaders  needed 
management  help  to  resolve  project  team  disagreements  several  times  [F4].  Usually  the 
team  knew  right  away  where  to  get  necessary  outside  help  [D16]. 
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Formal  reviews  were  conducted  at  key  decision  points  [D14].  The  primary  goal 
of  these  meetings  was  to  pass  high-level  data  among  the  key  players  and  the  Government. 
These  management  project  reviews  were  only  minimally  constructive  [D13].  These 
reviews  tended  to  take  away  from  the  flow  of  the  project  as  personnel  spent  extra  time 
with  the  rigid  documentation  requirements.  Reviews  for  major  weapon  systems  tend  to 
attract  large  numbers  of  participants.  Meetings  were  sometimes  unwieldy,  frustrating  and 
non-productive  [D7]. 

Later  in  development  team  members  started  to  go  to  the  shop  floor  to  meet  about 
related  production  processes  [FI].  Planning  meetings  were  held  that  included  both 
design  and  production  people  [FI  1  ] .  Physical  prototypes  were  passed  around  during 
these  joint  discussions  [F 10].  They  asked  for  supplier  comments  and  suggestions  on 
design  choices  [F2],  Team  members  showed  and  discussed  physical  models  of  new 
components  with  suppliers  [F3].  Design  and  production  technicians  explored  choices 
together  with  computational  models  and  analytical  tools  [F 12],  They  used  test  articles  or 
pre-production  hardware  to  discuss  and  examine  problems  [F13].  Just  prior  to  the 
production  transition  phase,  production  representatives  participated  regularly  in 
development  meetings  [Wl],  Team  members  also  began  to  meet  regularly  with 
production  personnel  out  on  the  shop  floor  [W2].  Technical  professionals  from 
production  started  to  have  unscheduled,  informal  joint  conversations  about  the  project 
with  design  personnel  [W16].  At  that  point,  analytic  engineering  tools  were  being  used 
jointly  by  design  and  production  [W17].  Prototypes  and  parts  were  being  used  regularly 
in  joint  discussions  [W18]. 

As  the  program  was  readied  for  production,  it  became  evident  that  logistics  skills 
were  lacking  from  the  program  [H3].  Realizing  the  deficiencies,  a  cross-functional 
working  arrangement  was  key  for  the  transition  into  production  [H2].  Logistics  is 
traditionally  pushed  off  until  the  end  of  the  program.  This  can  have  serious,  long  lasting 
effects  on  the  user  if  not  addressed.  Although  the  team  leader  was  technically  competent 
[D5],  he  had  little  experience  in  both  design  and  production  [D4].  By  the  time  the 
program  entered  production,  a  form  of  IPT  approach  was  used  to  resolve  problems  [H2]. 
Project  results  benefited  from  the  team’s  best  ideas  [D9], 
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F. 


RESEARCH  QUESTION  NUMBER  5:  KEY  PM  ISSUE 


What  was  the  key  issue  that  the  PM  had  to  deal  with  during  the  project  and  how 
was  it  dealt  with?  This  question  will  be  addressed  by  looking  at  the  primary  issue  where 
the  PM  focused  most  of  his  attention  and  resources. 

The  Government  survey  respondent  rated  control  of  the  production  project  as,  the 
biggest  fundamental  problem  the  PM  had  to  deal  with  in  managing  the  overall  program 
[12].  Problems  of  control  were  basically  the  external  environment,  i.e.,  the  sheer  number 
of  agencies  that  were  to  be  contended  with  on  a  regular  basis  under  the  “team”  approach. 
Like  the  internal  organization,  each  had  their  own  axe  to  grind  and  each  had  some  level 
of  input  and  “veto”  power.  As  an  example,  meetings  were  inordinately  large  and 
therefore  difficult  to  control.  Decisions  that  should  have  been  made  instantly  were 
negotiated  to  death  leaving  cost  and  schedule  impacts  to  be  resolved.  [12] 

The  PM  controlled  and  dictated  the  R&D  Program.  Had  he  not  dictated  the  R&D 
program  there  would  not  be  an  AAH  today  [12].  An  example  of  this  control  was  brought 
to  the  researcher’s  attention  during  a  Government  employee  interview.  This  occurred 
shortly  after  the  initial  production  contract  was  awarded  to  Hughes.  At  that  time,  Hughes 
Helicopters  was  headquartered  in  Culver  City,  CA.  Hughes  management  was  looking  for 
a  site  in  the  traditional  California  manufacturing  corridor  to  build  a  production  facility. 
Fearing  high  labor  costs  due  to  greater  competition,  the  Apache  PM,  General  Browne 
ordered  a  cost-analysis  study  of  the  area.  They  found  that  if  Hughes  located  in  this  high 
cost  area,  that  the  personnel  and  manufacturing  costs  could  reduce  the  total  Apache  buy 
nearly  in  half.  With  strong  urging  from  top  brass,  and  a  few  political  incentives,  Hughes 
chose  Mesa,  AZ  to  build  their  $300  million  facility  that  would  eventually  employ  two 
thousand  workers,  {ref  2} 
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V.  CONCLUSIONS 


A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  provide  conclusions  and  recommendations  for 
further  study.  I  will  address  significant  issues  found  in  the  analysis  that  helped  make  the 
program  a  success  or  that  caused  problems  or  delays.  Key  areas  from  question  1  are: 
user  representatives,  requirements,  and  funding.  Technology  readiness  from  question  2, 
testing  from  question  3  and  teaming  from  question  4  are  also  addressed  in  this  chapter. 


B.  LESSONS  LEARNED: 

User  Representatives: 

The  Apache  program  survived  in  a  difficult  political  climate  because  the  Program 
Manager  and  user  representative  worked  closely  together.  It  is  really  important  to  get  the 
user  representatives  on  board  early  and  it  is  most  beneficial  if  the  PM’s  relationship 
extends  to  form  a  close  working  relationship  with  the  user  community.  Good  user 
support  is  crucial  throughout  the  program,  and  including  the  user  in  all  major  reviews  can 
reinforce  it.  This  relationship  must  be  based  on  trust. 

Requirements: 

The  Apache  program  manager  kept  requirements  under  control  by  working 
together  with  the  user.  Requirements  creep  must  be  managed  but  can  be  kept  in  check  if 
stakeholders  have  a  clear  understanding  of  the  evolutionary  path  of  the  system.  With 
most  program  developments,  there  are  a  lot  of  contractors  who  will  try  to  sell  their 
systems  to  the  user.  The  PM  must  be  ready  to  manage  the  technological  side  of  the 
program  to  help  the  user  sort  through  the  “smoke  and  mirrors”  that  contractors  use  to 
hype  their  wares. 

Funding: 

The  Apache  program  experienced  several  funding  fluctuations;  the  PM  was  ready 
and  dealt  with  each  as  it  occurred.  Funding  stability  is  an  issue  in  any  large  program 
spread  out  over  many  years.  People  are  constantly  out  to  get  your  money.  You  need  to 
be  on  the  lookout  for  internal  suitors  from  your  own  service,  those  from  other  services 
and  outside  forces  from  Congress.  The  slightest  schedule  slip  or  problem  in  a  program 
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will  bring  its  competitors  to  its  doorstep  ready  to  take  funds  that  it  no  longer  can 
execute. 

Technology: 

The  Apache  program  had  several  changes  in  technology  throughout  development. 
They  were  able  to  track  technology  readiness  in  key  areas  and  mitigate  risk  by  moving 
certain  enhancements  off  the  critical  path.  Technology  readiness  also  played  a  vital  role 
when  adding  capabilities  such  as  the  Hellfire  missile.  The  technology  readiness  levels  of 
advanced  systems  must  be  clearly  articulated  to  the  user  by  the  PM  experts  to  assure  that 
the  users  requirements  can  be  met.  Technology  readiness  should  be  evaluated  throughout 
the  program  to  assure  that  the  system  can  stay  on  schedule.  Slips  in  technology  readiness 
can  greatly  cost  the  program. 

Teaming: 

The  Apache  program  was  developed  before  the  advent  of  formal  integrated 
product  teams  (IPT).  However,  survey  respondents  stated  that  a  form  of  IPT  was  used 
for  early  production.  Until  then,  the  program  was  put  together  in  smaller  pieces,  with 
teams  concentrating  solely  on  their  individual  area.  This  caused  delays  in  the  schedule 
when  key  components  were  not  ready  for  system  level  testing.  The  IPT  process  should 
be  utilized  to  assure  that  all  aspects  of  the  project  are  addressed.  Good  leadership  and  a 
clear  vision  are  keys  to  a  successful  IPT.  Membership  must  be  addressed  early  so  that 
decision  makers  are  consistently  present. 

Testing: 

The  Apache  program  manager  made  test  readiness  a  primary  goal.  The  test  team 
was  properly  staffed  with  the  proper  resources  at  their  disposal.  The  test  plan  is  an 
important  document  that  helps  lay  out  the  program  schedule.  By  performing  a  Failure 
Modes  Effects  and  Criticality  Analysis  (FMECA)  early,  the  Apache  PM  was  able  to  use 
the  results  to  help  build  the  test  plan.  This  information  also  feeds  into  the  Test 
Evaluation  Master  Plan  (TEMP)  required  for  Milestone  reviews.  Testing  on  the  Apache 
followed  a  traditional  approach  of  test-fix-test.  The  system  had  clear  transitions  from 
development  to  operational  testing.  The  test  plan  was  modified  with  funding  and 
schedule  slips.  It’s  the  program  manager’s  job  to  make  sure  that  the  system  is  ready  for 
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test.  In  the  end,  the  fact  that  the  system  was  able  to  demonstrate  its  operational  capability 
in  real  world  environments  helped  save  the  program  from  cancellation. 

The  survey  respondents  documented  several  operational  problems.  It’s 
impractical  to  test  out  every  potential  operational  scenario.  Unforeseen  problems  and 
systems  deficiencies  are  found  nearly  every  time  a  new  system  is  fielded. 


C.  AREAS  FOR  FURTHER  RESEARCH 

This  thesis  addressed  mission  equipment  systems  integrated  into  the  Apache 
Attack  Helicopter.  There  were  several  other  Apache -related  technologies  that  could  be 
addressed  by  future  studies.  Possible  areas  of  study  might  include:  engines,  30  mm  gun, 
and  advanced  materials. 

Funding:  Evaluate  the  effects  of  delaying  funding  and  stretching  the  schedule. 
Another  potential  area  is  the  overall  effect  of  quantity  cuts  on  the  per-unit  cost  of  major 
weapon  systems. 

Testing:  Evaluate  the  traditional  test  methods  of  separate  development  test  and 
operational  test  versus  the  combined  testing  philosophy  used  today.  Does  the  new 
method  really  save  time  and  money  in  a  large  program? 

Readiness:  Evaluate  the  use  of  Technology  Readiness  Levels  developed  and  used 
in  the  Air  Force. 

Leadership:  Evaluate  the  effectiveness  of  civilian  leadership  in  the  early  stages  of 
a  complex  acquisition  category  AC  AT  ID  program. 
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APPENDIX  C:  COMBINED  QUESTIONNAIRE 


DESERT  STORM  CASE  STUDY  CHECKLIST: 

LESSONS  FOR  TECHNOLOGY  MANAGEMENT 

The  U.S.  Army  Materiel  Command  is  supporting  a  hindsight  study  of  how  technologies  were 
developed,  integrated  into  systems,  and  produced  in  the  years  leading  up  to  Desert  Storm,  the  last  large- 
scale  deployment  of  U.S.  military  force.  It  is  believed  that  in  the  years  leading  up  to  that  conflict,  there 
were  both  successful  and  unsuccessful  applications  of  technology  to  military  systems  that  contain  lessons 
for  future  defense  technology  development.  The  study  can  be  done  now  because  the  intervening  years 
allow  more  objectivity,  and  allow  open  examination  of  what  were  once  classified  projects.  The  study 
must  be  done  now  because  many  of  the  men  and  women  responsible  for  the  development  and  eventual 
fielding  of  those  systems  in  Gulf  region  are  retiring,  taking  with  them  important  knowledge  that  we 
believe  should  be  captured  and  codified  into  practical  lessons  for  the  future. 

Our  method  began  with  a  list  of  military  systems  including  both  successes  and  failures  judged  to 
be  broadly  representative  of  the  systems  that  were  under  development  in  the  years  prior  to  Desert  Stonn. 
Then  experienced  students  (such  as  those  found  at  senior  military  schools  and  mid-career  management 
programs)  are  being  asked  to  create  a  single  case  study  for  a  project  on  that  list.  Each  case  will  include 
both  (1)  a  narrative  case  history  to  capture  the  richness  of  the  case  and  identify  any  factors  that 
determined  a  project’s  success  or  failure,  and  (2)  answers  to  structured  questions  that  ask  about 
organization,  technology  and  process  issues  in  a  consistent  way  across  all  cases. 

Participants  in  the  selected  projects  are  being  asked  to  complete  this  survey  form  as  background 
information  for  the  students  to  use  in  their  projects,  and  we  hope  you  can  cooperate  with  our  research. 

This  is  not  a  traditional  questionnaire.  If  you  do  not  remember  the  details  we  are  asking  about,  or 
if  you  feel  that  the  answer  would  be  misleading  or  somehow  inappropriate  for  the  project  we  are  asking 
you  about,  feel  free  to  leave  the  answer  blank.  Y ou  may  rewrite  the  question  so  it  fits  better.  If  you  have 
comments  to  add,  or  want  to  suggest  a  better  answer  than  what  is  provided,  feel  free  to  do  so. 

While  the  students  conducting  this  research  may  be  cleared  to  discuss  classified  material,  it  should 
be  stressed  that  the  narratives  and  the  answers  to  structured  questions  should  never  include  any  classified 
information.  The  results  will  be  used  in  unclassified  reports. 
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You  may  request  a  copy  of  any  report  of  the  findings  by  providing  your  business  card,  or 
providing  a  separate  sheet  of  paper  with  your  name  and  address  information,  including  your  e-mail 
address.  If  you  have  any  questions,  contact: 

Richard  G.  Rhoades  William  A.  Lucas 

Research  Institute  Sloan  School  of  Management 

The  University  of  Alabama  in  Huntsville  Massachusetts  Institute  of  Technology 

Rhoadesr@email.uah.edu  (256)  824-6343  walucas@mit.edu  (617)253-0538 

To  begin:  The  first  set  of  questions  defines  three  dates,  keyed  to  technology  readiness  levels  (see  last 
page),  and  then  asks  about  the  roles  played  by  different  organizations  at  three  stages  of  your  project 
leading  up  to  those  dates.  The  organizations  of  interest  are: 

Prime’s  S&T  org.:  Group  within  system  prime  contractor  responsible  for  doing  IR&D  and 

developing  new  technology  and  concepts. 

Other  prime  org.:  Any  prime  contractor  organization  other  than  the  S&T  organization. 

Supplier  S&T:  Same  definition  as  for  prime’s  S&T  organization,  but  located  at  a  supplier. 

Other  supplier  org.:  Any  supplier  organization  other  than  the  S&T  organization. 

Army  Lab/Center:  One  or  more  of  the  Army  laboratories  or  research,  development  and  engineering 

Centers. 

Other  DoD/S&T  org.:  An  equivalent  of  an  Army  Lab/Center  found  elsewhere  in  DoD. 
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SP.  What  was  the  approximate  starting  date  of  systems  planning  and  pre-development  work?  This  date  is  when 
planning  work  began  on  the  integrated  system.  The  systems  concept  and  applications  had  been  formulated, 
but  applications  were  still  speculative.  There  was  no  proof  or  detailed  analysis  to  support  the  approach. 

SYSTEMS  PLANNING  START  DATE  (SP):  _ January  /  1970  (mo/yr)  [TRL2  at  system  level] 

In  what  organization  was  the  primary  work  leading  up  to  this  point  accomplished?  CHEYENNE  PM,  with  lead 
support  from  TRADOC. 

Including  that  organization,  what  organizations  had  been  involved  up  to  this  point?  (Check  the  role  of  each) 


Prime’s  S&T  org. 

Lead/co-lead 

_ Active  support 

_ Involved 

Kept  informed 

X  Not  involved 

_ DK 

Other  prime  org. 

Lead/co-lead 

X  Active  support 

_ Involved 

Kept  informed 

Not  involved 

_ DK 

Supplier  S&T  org. 

Lead/co-lead 

_ Active  support 

X  Involved 

Kept  informed 

Not  involved 

_ DK 

Other  supplier  org. 

Lead/co-lead 

X  Active  support 

_ Involved 

Kept  informed 

Not  involved 

_ DK 

Army  Lab/Center 

X  Lead/co-lead 

_ Active  support 

Involved 

Kept  informed 

Not  involved 

_ DK 

Other  DoD/S&T  org. 

Lead/co-lead 

X  Active  support 

Involved 

Kept  informed 

Not  involved 

_ DK 

What  was  the  nature  of  the  Army  Lab/Center’s  involvement?  (Simulation?  Concept  formulation?  Integration? 
Requirements  development?)  Most  of  the  behind  the  scene  ‘effort’  was  competing  contractors  (Bell  and  Sikorsky), 
supporting  their  King  Cobra  and  S-67  as  replacement  for  the  LOCKHEED  CHEYENNE  while  supporting  termination  of 
the  CHEYENNE  Production  contract  as  well  as  the  eventual  development  contract.  Definition  of  the  Advanced  Attack 
Helicopter  was  centered  around  what  the  Department  of  Army  thought  the  helicopter  industry  was  capable  of  producing. 

D.  Date  when  Development  started.  Typically  at  this  date,  funding  started  for  system  advanced  or  engineering 
development,  a  gov’t  project  office  was  formed  and  prime  contractor(s)  selected. 

DEVELOPMENT  START  DATE:  (D):  April  /  1973  (mo/yr) 

What  was  the  Technology  Readiness  Level  (refer  to  page  8)  for  the  SYSTEM  on  this  date?  5:  Components 

and/or  bread  board  validation  in  relevant  environment.  Fidelity  of  breadboard  technology  is  significantly  increased.  Basic 

components  integrated  with  reasonably  realistic  supporting  elements  so  the  technology  can  be  tested  in  a  simulated 

environment.  Examples  include  “high  fidelity”  laboratory  integration  of  components. 

What  was  the  Production  Readiness  Level  (refer  to  page  8)  for  the  SYSTEM  on  this  date?  2:  The  application  is 

produced  outside  the  lab  with  tools  and  processes  used  for  producing  very  low  quantities. 

In  what  organization  was  the  primary  work  in  the  period  from  SP  to  D  accomplished?  Army  Lab  /  PM 

Including  that  organization,  what  organizations  had  been  involved  in  the  period  SP  to  D?  (Check  the  role  of  each.) 


Prime’s  S&T  org 

Lead/co-lead 

_ Active  support 

Involved 

Kept  informed 

X  Not  involved 

_ DK 

Other  prime  org 

Lead/co-lead 

_ Active  support 

X  Involved 

Kept  informed 

Not  involved 

_ DK 

Supplier  S&T 

Lead/co-lead 

_ Active  support 

X  Involved 

Kept  informed 

Not  involved 

_ DK 

Other  supplier  org 

Lead/co-lead 

_ Active  support 

X  Involved 

Kept  informed 

Not  involved 

_ DK 

Army  Lab/Center 

X  Lead/co-lead 

_ Active  support 

Involved 

Kept  informed 

Not  involved 

_ DK 

Other  DoD/S&T  org. 

Lead/co-lead 

X  Active  support 

Involved 

Kept  informed 

Not  involved 

_ DK 

What  was  the  nature  of  the  Army  Lab/Center’s  involvement?  (Engineering  support?  Simulation  or  testing? 
Integration?  Requirements  interpretation?)  Engineering  support  &  Requirements  interpretation 


TP.  Date  of  achieving  “Transition  to  Production”  when  producible  system  prototype  has  been  demonstrated  in  an 

operational  environment.  Prototype  is  near  or  at  planned  operational  system,  produced  on  small  scale.  TRANSITION 
TO  PRODUCTION  (TP)  DATE:  April  /  1982  (mo/yr)  (TRL7  at  system  level) 

What  was  the  Production  Readiness  Level  for  the  SYSTEM  on  this  date?  3:  The  application  is  produced  in  low 
quantities  with  tools  and  processes  planned  to  be  used  in  production  systems.  Testing  procedures  for  components 

and  subsystems  are  established. 


In  what  organization  was  the  primary  work  in  the  period  from  D  to  TP  accomplished?  Prime  S&T  Organization 
Including  that  organization,  what  organizations  had  been  involved  in  the  period  D  to  TP?  (Check  the  role  of  each.) 


Prime’s  S&T  org 

X  Lead/co-lead 

Active  support 

_ Involved 

Kept  informed 

Not  involved 

_ DK 

Other  prime  org 

Lead/co-lead 

Active  support 

_ Involved 

Kept  informed 

Not  involved 

_ DK 

Supplier  S&T 

Lead/co-lead 

JC 

Active  support 

Involved 

Kept  informed  _ 

Not  involved 

_ DK 

Other  supplier  org 

Lead/co-lead 

Active  support 

X  Involved 

Kept  informed 

Not  involved 

DK 
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What  was  the  nature  of  the  Army  Lab/Center’s  involvement?  (Engineering  support? 
Simulation  or  testing?  Integration?  Requirements  interpretation?)  Engineering  support.  Simulation  & 
Testing 


Please  note:  Here  we  shift  away  from  the  system  as  a  whole,  and  move  to  its  component 
technologies. 

T1 .  Now  identify  one  or  more  (up  to  3)  technologies  that  were  incorporated  into  the  system  you  are 

studying.  These  technologies  should  be  among  those  central  to  the  success  of  the  system. 

Technology  A  Target  Acquisition/Designation/Pilot  Night  Vision  System  (TADS/PNVS) 
Technology  B  Computers 

Technology  C  Avionics 

T2.  How  new  was  each  technology  to  the  prime  contractor?  For  each  technology  A,  B,  and  C,  was  the 
technology:  (Answer  for  date  you  provided  for  Development  start ,  D.) 


Technology  A 

Technology  B 

Technology  C 

TADS/PNVS 

Computers 

Avionics 

New  and  unproven  for  the  prime  contractor? 

1.  X_ 

1.  X_ 

1.  X 

Technology  had  been  used  bv  prime  contractor  but  it  was 

to  this  kind  of  application? 

new  to 

2.  _ 

2.  _ 

2.  _ 

Technology  had  been  used  by  prime  contractor  in  similar 

and  was  well  understood? 

applications 

3.  _ 

3.  _ 

3.  _ 

Don’t  know,  can’t  remember,  or  would  have  to  guess 

9.  _ 

9.  _ 

9.  _ 

T3.  Production  Impact.  What  was  the  impact  of  the  technology  on  then  existing  production  processes? 

(Answer  for  date  you  provided  for  Development  start ,  D.) 


Technology  A 

TADS/PNVS 

Technology  forced  deep  and  serious  production  process  change?  1 .  _ 
Technology  caused  significant  production  process  change?  2.  X 
Technology  did  not  require  much  production  process  change  3.  _ 

Don’t  know,  can’t  remember,  or  would  have  to  guess  9. _ 


Technology  B 

Computers 
1.  X_ 

2. _ 

3. _ 

9. 


Technology  C 

Avionics 
1. _ 

2.  X 

3.  _ 

9. 


T4.  Looking  back  at  the  Development  start  date,  at  that  time  how  important  were  these  technologies  to 
the  prime? 

Check  (V)  the  best  answer  for  each  technology.  Technology  A  Technology  B  Technology  C 


TADS/PNVS 

This  system  was  the  Prime’s  only  planned  application  of  the  technology.  1 .  X 
Prime  was  planning  or  had  started  follow-on  uses  of  the  technology.  2.  _ 
Technology  was  being  used  in  other  applications  and  it  was 
expected  to  be  significant  area  of  competence  for  the  Prime.  3.  _ 

Don’t  know,  can’t  remember,  or  would  have  to  guess.  9.  _ 


Computers  Avionics 

1. _  1. _ 

2.  X  2.  X 


3. 

9. 
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Now  look  at  the  SP,  D,  and  TP  dates  you  provided  above  for  the  System.  Using  the  Technology  Readiness  (TRL) 
Scale  on  page  8,  find  the  number  that  represents  the  readiness  of  the  separate  technologies  the  team  was  working 
with  at  each  point  in  time.  Please  answer  here  for  the  state  of  development  of  each  component  technology.  (NOT 
for  the  over-all  system  which  was  the  focus  in  the  questions  above.) 

Technology  A  Technology  B  Technology  C 

TADS/PNVS  Computers  Avionics 

T5.  When  System  planning  and  pre-development  began,  technology  TRL  was:  # _ 2 _  # _ 2 _  # _ 6 _ 

T6.  When  System  went  into  Development,  technology  TRL  was:  # _ 4 _  # _ 4 _  # _ 7 _ 

T7.  When  System  reached  Transition  to  Production,  technology  TRL  was:  # _ 8 _  # _ 8 _  # _ 8 _ 


T8.  For  each  of  the  technologies  A,  B  &  C,  did  an  Army  Laboratory  or  Center  make  a  significant  contribution  to 
achieving  any  of  the  above  levels  of  technology  readiness? 


T8.  Yes,  it  contributed  to  Readiness  at  start  of  Planning/Pre-development. 
T9.  Yes,  it  contributed  to  Readiness  for  Development. 

T10.  Yes,  it  contributed  to  Readiness  for  Transition  to  Production. 

Tn.  No,  an  Anny  lab  or  center  did  not  make  a  significant  contribution. 
Tdk.  Don’t  know,  can’t  say,  don’t  remember. 


Technology  A 

TADS/PNVS 

8a  Y_ 

9a.  Y_ 
10a.  Y_ 

No  a _ 

DKa. 


Technology  B  Technology  C 


Computers  Avionics 


8b.  _ 

9b.  _ 

10b.  Y 

No-b _ 

DKb 


8c.  Y 
9c.  Y 
10c.  Y 

No-c _ 

DKc 
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Project  History,  Staffing  and  Location 

HI.  At  some  point,  was  the  project  either:  X  1.  Slowed  down?  _ 2.  Stopped  and  restarted?  _ 3.  Neither 

H2.  Was  the  project  set  up  as  a  cross-functional  integrated  product  team  (IPT),  a  project  team  drawn  from  different 

parts  of  the  contractor’s  organization  with  most  of  the  skills  needed  for  the  development?  _ Yes  X _ No 

If  YES,  was  it:  _  1.  Set  up  by  management,  with  different  functions  &  departments  tasked  to  provide  team  members. 

_  2.  Set  up  informally,  with  team  expected  to  ask  departments  for  help  as  needs  emerged. 

H3.  Key  Skills.  This  question  asks  about  “key  skills”  essential  to  the  success  of  the  project,  defined  as  skills 

“that  if  they  were  not  available  at  all,  would  have  stopped  team  progress  at  the  point  when  they  were  needed.” 

Were  there  any  key  skills  not  adequately  represented  on  the  team?  _ No.  X  Yes,  one.  _ Yes,  more  than  one. 

IF  YES:  H35.  What  were  the  missing  key  skills?  Please  check  (/  )  any  and  all  that  apply. 

_  1.  Internal  technical  professionals  _ 4.  Technical/development  people  from  Suppliers 

_  2.  Producibility  professionals  (DFM,  other)  _ 5.  Producibility  professionals  from  Suppliers 

_  3.  Financial/contracts  professionals  X  6.  Other.  Please  specify  Logistics 

H4.  During  the  Development  stage  of  the  project,  how  many  people  on  the  team  were  collocated  very  close 
together?  (On  the  same  floor  of  a  building  within  a  one  minute  walk.) 

_ 1.  All  _ 2.  Most  (2/3rds  or  more)  _ 3.  Some  (over  a  third)  X  4.  Few  _ 5.  None  _ DK/can’t  say 

H4a.  Including  the  above,  how  many  people  on  the  team  were  collocated  in  the  same  building? 

_ 1.  All  _ 2.  Most  (2/3rds  or  more)  X  3.  Some  (over  a  third)  _ 4.  Few  _ 5.  None  _ DK/can’t  say 

H5.  How  many  people  on  the  team  involved  in  the  Development  stage  had  worked  before  with  others  on  the  project? 

_ 1 .  All  _ 2.  Most  (2/3rds  or  more)  X  3 .  Some  (over  a  third)  _ 4.  Few  _ 5.  None  _ DK/can’t  say 

H6.  Whose  facilities  were  going  to  be  the  primary  production  site  for  the  application  of  the  new  technologies? 

_ 1.  Prime  contractor’s  facilities  X  2.  Both  prime  and  supplier  facilities  _ 3.  Supplier  facilities 

Validation  Activities:  Testing  and  Simulation 

VI .  Was  a  failure  modes  and  effects  analysis  done  on  the  system?  1  X  Yes  2. _ No  3. _ Don’t  know 

Via.  If  yes,  was  it  used  to  help  establish  the  test  plan?  1.  X  Yes  2. _ No  3.  _ Don’t  know 

For  individual  components: 

V2.  Was  there  testing  to  see  if  the  individual  components  of  the  system  worked?  What  organization) s)  did  this  testing? 

X  Prime  X  Suppliers  X  Army  center/lab  _ Other  govt  org.  _ Not  done  on  project  _ Don’t  know 

V3.  Were  there  simulations  run  to  see  if  the  individual  components  of  the  system  worked?  What  organization) s) 
did  these  simulations? 

X  Prime  X  Suppliers  X  Army  center/lab  _ Other  govt  org.  _ Not  done  on  project  _ Don’t  know 

For  integrated  components  in  controlled  setting: 

V4.  Were  the  components  tested  working  together  in  a  controlled  setting?  What  organization(s)  did  this  testing? 

X  Prime  X  Suppliers  X  Army  center/lab  _ Other  govt  org.  _ Not  done  on  project  _ Don’t  know 

V5.  Were  there  simulations  of  the  components  working  together  in  a  controlled  setting?  What  organization(s)  did  this? 
X  Prime  X  Suppliers  X  limited  Army  center/lab  _ Other  govt  org.  _ Not  done  on  project  _ Don’t  know 

For  integrated  components  in  a  realistic  setting: 

V6.  Was  there  testing  of  the  components  working  together  in  a  realistic  setting?  What  organization(s)  did  this  testing? 

X  Prime  X  Suppliers  X  Army  center/lab  X  Other  govt  org.  _ Not  done  on  project  _ Don’t  know 

V7.  Was  a  hardware-in-the-loop  type  systems  integration  simulation  laboratory  used? 

V7a.  To  see  if  the  individual  components  of  the  system  worked:  X  l.Yes  _ 2.  No  _ 9.  Don’t  know 

V7b.  To  see  if  integrated  components  worked  in  controlled  setting:  X  l.Yes  _ 2.  No  _ 9.  Don’t  know 
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V8.  Recalling  the  total  effort  (100%)  spent  on  testing  and  simulations,  please  allocate  the  percent  of  that  total  that  were: 

_ 10  %  spent  to  see  if  the  individual  components  of  the  system  worked 

_ 65  %  spent  to  see  if  integrated  components  worked  in  controlled  setting 

_ 20  %  spent  to  see  if  integrated  components  worked  in  a  realistic  setting 

05  %  spent  on  any  other  validation  purpose. 

100  % 


Please  evaluate  the  following  statements  Strongly 

about  the  use  of  testing  and  simulations  on  the  project.  disagree 

Disagree 

somewhat 

Neither  agree  Agree 
nor  disagree  somewhat 

Strongly  Don’t 
agree  know 

V9.  Knowledge  from  validation  work  was  used  consistently 

to  improve  components  and  system.  1 

2 

3 

X 

5 

9 

V10.  Project  test  philosophy  was  to  “Break  it  big  early.”  1 

X 

3 

4 

5 

9 

VI 1.  Component  and  system  maturity  were  validated  at  the  right 

times  in  the  program.  1 

2 

X 

4 

5 

9 

V12.  The  project  and  the  testing  community  had  an  adversarial 

relationship.  1 

X 

3 

4 

5 

9 

V13.  Most  project  validation  events  produced  quality  results.  1 

2 

X 

4 

5 

9 

V14.  The  project  didn’t  recognize  important  lessons  that  validation 

work  uncovered.  1 

2 

X 

4 

5 

9 

VI 5.  Sometimes  the  project  settled  for  less  than  the  best 

validation  method.  1 

2 

X 

4 

5 

9 

Team  Participants  &  Communications  during  Development 

Here  are  some  statements  about  the  people  on  the  project  during  the  System  Development  stage.  Please  circle  a  number  to 
indicate  your  level  of  agreement  or  disagreement  that  each  statement  is  a  description  of  team  processes  on  this  project. 

Strongly  Disagree  Neither  agree  Agree 
disagree  somewhat  nor  disagree  Somewhat 

Strongly 

Agree 

Don’t 

know 

D1 .  The  team  leader  was  good  at  resolving  technical  disagreements.  1 

2 

3 

X 

5 

9 

D2.  The  team  leader  was  good  at  getting  necessary  resources.  1 

2 

3 

X 

5 

9 

D3.  There  was  a  lot  of  turn-over  in  team  membership.  1 

X 

3 

4 

5 

9 

D4.  The  team  leader  had  both  design  &  production  experience.  1 

X 

3 

4 

5 

9 

D5.  The  team  leader  had  verv  high  technical  competence.  1 

2 

3 

X 

5 

9 

D6.  Some  key  technical  skills  were  not  represented  on  the  team  itself.  1 

X 

3 

4 

5 

9 

D7.  Team  meetings  were  sometimes  frustrating  and  non-productive.  1 

2 

3 

X 

5 

9 

D8.  Professionals  were  split  across  too  many  different  tasks  &  teams.  1 

2 

X 

4 

5 

9 

D9.  Project  results  did  not  take  advantage  of  the  team’s  best  ideas.  1 

X 

3 

4 

5 

9 

D10.  Key  members  continued  through  pre-production  planning 

and  testing.  1 

2 

3 

X 

5 

9 

Dll.  There  was  often  uncertainty  about  the  future  of  project  funding.  1 

2 

3 

X 

5 

9 

D12.  The  team  was  reluctant  to  share  concerns  with  gov’t  PM.  1 

X 

3 

4 

5 

9 

D13.  Management  project  reviews  were  constructive  &  helpful.  1 

2 

X 

4 

5 

9 

D14.  Formal  reviews  were  conducted  at  key  decision  points.  1 

2 

3 

4 

X 

9 

D15.  At  the  prime  contractor,  the  project  was  a  management  priority.  1 

2 

3 

4 

X 

9 

D16.  Usuallv  team  knew  right  away  where  to  get  necessary  outside  help.  1 

2 

3 

X 

5 

9 

D17.  Project  had  a  visible  &  supportive  champion  in  the  Prime’s  management.  1 

2 

3 

4 

X 

9 

D 18. There  was  a  lot  of  contact  with  TRADOC*  during  the  project.  1 

2 

3 

4 

X 

9 

D19.  The  gov’t  PM  was  reluctant  to  share  problems  with  Army  leaders  1 

2 

X 

4 

5 

9 

*  By  TRADOC  here  and  elsewhere,  we  mean  Training  &  Doctrine  Command  and/or  other  appropriate  user  representatives. 
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D20.  Who  besides  the  team  usually  attended  formal  reviews?  (Check  all  that  apply.) 


D20a.  Any  Prime  upper  management  (Director  or  VP  level)? 

X  1.  Yes 

_ 2.  No 

_ 8.  Not  appl. 

_ 9.  DK 

D20b.  Any  Army  Program  management  representatives? 

X  1.  Yes 

_ 2.  No 

_ 8.  Not  appl. 

_ 9.  DK 

D20c.  Any  TRADOC  or  other  user  representatives? 

X  1.  Yes 

_ 2.  No 

_ 8.  Not  appl. 

_ 9.  DK 

Activity  Report  during  System  Development  Stage  of  Project 
How  often  did  team  members  do  the  following  during  Development?  Once  or  Several  Many  Don’t  know, 


(If  you  feel  the  activity  is  Not  Applicable  to  your  project,  check  NA.)  Never  twice  times  times  Not  appl 

FI.  Went  to  the  shop  floor  to  meet  about  related  production  processes.  X  ( _ ) 

F2.  Asked  for  supplier  comments  &  suggestions  on  design  choices.  _X _  ( _ ) 

F3.  Showed  &  discussed  physical  models  of  new  components  with  suppliers.  _  _X _  ( _ ) 

F4.  Needed  management  help  to  resolve  project  team  disagreements.  _X _  ( _ ) 

How  often  did  the  following  occur  during  Development? 

F5.  Did  TRADOC/other  user  organizations  show  strong  support?  _  _  _  _X _  ( _ ) 

F6.  Were  there  changes  in  key  TRADOC  or  other  user  personnel?  _  _  _X _  ( _ ) 

F7.  Were  there  changes  in  system  requirements  (e.g.,  threat)?  _  _  _X _  ( _ ) 


SHARED  DESIGN-PRODUCTION  ACTIVITIES  during  System  Development.  Here  only  count  joint  meetings  or 
discussions  that  included  both  DESIGN  and  people  from  PRODUCTION 

and/or  from  the  PROGRAM  concerned  with  production  of  the  System.  Once  or  Several  Many  Don’t  know, 


How  often  did  team  members  do  the  following  during  Development?  Never  twice  times  times  Not  appl 

F10.  Passed  around  physical  prototypes  during  joint  discussions.  X _  ( _ ) 

F 1 1 .  Held  planning  meetings  that  included  both  design  &  production  people.  _  _X _  ( _ ) 

F12.  Explored  choices  together  with  computational  models  or  analytic  tools.  _  _X _  ( _ ) 

F13.  Had  test  articles  or  pre-production  parts  to  discuss  and  examine  jointly.  _  _X _  ( _ ) 


SHARED  DESIGN-SUPPLIER  ACTIVITIES  during  System  Development.  Now  only  count  joint  meetings  or 
discussions  that  included  personnel  from  both  DESIGN  and  SUPPLIERS. 

How  often  did  team  members  do  the  following  during  Development?  Once  or  Several  Many  Don’t  know, 

Never  twice  times  times  Not  appl. 


F20.  Passed  around  physical  prototypes  during  joint  discussions.  X _  ( _ ) 

F21.  Held  planning  meetings  that  included  both  design  and  suppliers.  _X _  ( _ ) 

F22.  Explored  choices  together  with  computational  models  or  analytic  tools.  _ X _  ( _ ) 

F23.  Had  test  articles  or  pre-production  parts  to  discuss  and  examine  jointly. _  X _  ( _ ) 


ACTIVITY  PHASING  BY  STAGES  OF  DEVELOPMENT  AND  TRANSITION 
When  were  the  following  activities  carried  out  by  the  team?  For  example,  if  for  W 1,  Production  was  involved  regularly 
in  the  Selection/Planning  stage,  dropped  out,  and  then  came  back  in  late  in  the  Development  work  and  continued  to  participate 
after  that,  check  (V)  first,  fourth  and  fifth  columns. 

SP  D  TP 

|  Selection  |  Development  I  Transition  (DK/ 

Please  check  (V  )  all  stages  when  the  activity  occurred.  ▼  ^arly  Middle  Later  ^  (Never!  NA) 


Wl.  When  did  production  representatives  participate  regularly? 

W2.  When  did  team  members  meet  with  production  on  shop  floor? 

W3.  When  was  the  TRADOC  consulted  on  project  questions? 

X 

X  (  )(  ) 

X 

X 

XXX 

X  ( )  ( ) 

X  (  )(  ) 

W4.  When  was  there  change  in  key  TRADOC/user  representatives? 

W5.  When  did  TRADOC/other  users  show  strong  support? 

W6.  When  was  there  change  in  the  system  requirements? 

X 

X  X 

jL  ( _ )  ( _ ) 

X  (__)() 

X 

X  ( _ )( _ ) 
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RelationshiD  &  Activities  between  Enqineerinq  Desiqn  &  Prod 

These  questions  are  different  because  they  focus  only 
on  joint  meetinqs  or  discussions  that  included  both  SP 

DESIGN  personnel  and  people  from  PRODUCTION 
and/or  PROGRAM  people  concerned  with  production.  ^ 

W16.  When  did  the  team  &  technical  professionals  from  Production 
have  unscheduled  &  informal  joint  conversations  about  the  project? 

uction/Pr 

D 

Selection 

ogram 

TP 

Development 

J  Early  Middle  Later  ^ 

X 

Transition  (DK/ 

J  (Never)  NA) 

(  )  (  ) 

W17.  When  were  analytic  engineering  tools  used  jointly  bv  Design 
and  Production  to  explore  options  together? 

X 

(  )  (  ) 

W18.  When  were  prototypes  and  parts  used  in  joint  discussions? 

Relationship  &  Activities  between  Enqineerinq  Desiqn  & 

Supplie 

X 

rs 

(  )  (  ) 

ransition  (DK/ 

Focus  onlv  on  joint  meetinqs  or  discussions  that  S 

included  both  DESIGN  personnel  and  SUPPLIERS: 

W26.  When  did  the  team  &  technical  professionals  from  Suppliers  1 
have  unscheduled  &  informal  joint  conversations  about  the  project? 

»  D 

Selection 

TP 

Development 

farly  Middle  Later  ' 
X 

^  (Never)  NA) 

(  )  (  ) 

W27.  When  were  analytic  engineering  tools  used  jointly  bv  Design 
and  Suppliers  to  explore  options  together? 

X 

(  )  (  ) 

W28.  When  were  prototypes  and  parts  used  in  joint  discussions? 

X 

(  )  (  ) 

Problem  Solving  and  Team  Effort 

Here  are  a  series  of  statements  about  problems  that  are  said  to  occur  with  technology  development.  For  each  statement,  we  are 

asking  you  to  make  two  separate  judgments  to  help  us  understand  what  problems  require  substantial  team  effort: 

O  First,  did  this  problem  ever  come  up  in  the  specific  project  being  reported  on?  If  “No”,  then  circle  the  “0”. 

©  If  “Yes,”  how  serious  was  the  impact  of  this  problem  on  the  process  of  the  project’s  work?  Here  we  are  concerned  with 
how  much  effort  in  attention,  time  and  energy  did  the  project  have  to  spend  solving  or  compensating  for  this  problem. 


O  Did  this  problem  come  up  during  this  project? 

No. 

Yes.  The  problem  came  up. 

B 1 .  It  was  harder  than  expected  to  take  the  risk  out  of  the  new  technology. 

0 

©  IF  YES.  how  much  project  effort 

had  to  be  snent  on  this  problem? 

Very  Very 

minor  Minor  Signif.  Major  major 
effort  effort  effort  effort  effort 

1X345 

B2.  Cut-backs  in  project  resources  forced  changes/compromises. 

0 

1 

2 

X 

4 

5 

B3.  Changes  in  company  strategies  and  goals  hurt  the  project. 

0 

1 

2 

X 

4 

5 

B4.  A  critical  production  issue  was  uncovered  very  late  in  the  process. 

0 

1 

X 

3 

4 

5 

B5.  Management  pressure  pushed  technology  prematurely  into  production. 

0 

1 

2 

X 

4 

5 

B6.  There  was  a  lack  of  acceptance  standards  for  the  new  technology. 

0 

1 

X 

3 

4 

5 

B7.  The  technology  was  hard  to  scale  up  from  lab  &  pilot  tests. 

0 

1 

X 

3 

4 

5 

B8.  Testing,  quality  control  and/or  acceptance  took  longer  than  planned. 

0 

1 

2 

X 

4 

5 

B9.  Departments  at  the  prime  resisted  project  ideas  &  approaches. 

0 

1 

2 

X 

4 

5 

BIO.  One  or  more  suppliers  did  not  meet  their  commitments. 

0 

1 

X 

3 

4 

5 

Bll.  Army  Labs/Centers  resisted  project  ideas  or  approaches. 

0 

1 

X 

3 

4 

5 

B12.  Army  program  offices  resisted  project  ideas  or  approaches. 

0 

1 

X 

3 

4 

5 

B13.  Threat  definition  or  other  requirements  changed  during  the  project. 

0 

1 

2 

X 

4 

5 
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Project  Outcomes 

01.  Project  Acceptance.  Was  the  SYSTEM  accepted  to  be  put  into  Production?  This  is  initial  acceptance,  not  whether  it 
actually  ended  up  in  production. 

1.  No,  the  System  was  abandoned.  _  8.  NA,  not  applicable 

_  2.  No,  but  concept/technology  was  used  later.  _  9.  DK,  don’t  know/can’t  remember 

X  3.  Yes,  the  System  was  accepted  for  production 


02.  After  the  SYSTEM  was  accepted  and  was  in  Transition  to  Production,  how  many  additional  changes  in  the  designs  and 
processes  were  later  required  before  the  System  was  taken  into  full  production? 

_  1 .  Many  serious  changes  X  2.  Significant  changes  _ 3.  Minor  changes  _ 4.  No  or  almost  no  changes 

7.  Did  not  reach  production,  was  not  implemented  _ 8.  Not  Applicable  _ 9.  Don’t  know 


03.  Did  the  SYSTEM  go  into  full  production? 

1.  No,  the  System  was  abandoned.  _  8.  NA,  not  applicable 

_  2.  No,  but  concept/some  technology  was  used  later.  _  9.  DK,  don’t  know/can’t  remember 


X  3.  Yes,  the  System  was  put  into  full  production. 

04.  For  each  of  the  technologies  A,  B,  and  C  above,  to  what  04a.  04b. 

extent  was  each  used  in  the  System  as  it  was  produced:  Technology  A  Technology  B 

No,  the  technology  was  not  used  in  the  System.  1 . _  1 . _ 

No,  but  the  technology  was  used  later(elsewhere)  2. _  2. _ 

The  technology  was  used  but  not  to  extent  originally  planned.  3. _  3. _ 

Yes,  the  technology  was  used  as  planned.  4.  X  4.  X 

Don’t  know.  9.  9. 


04c. 

Technology  C 

1. _ 

2. _ 

3.  _ 

4.  X 
9. 


05.  After  the  SYSTEM  reached  Transition  to  Production^  did  the  project  go  to  Production  as  quickly  as  it  should  have? 

1 .  No  delay  X  2.  One  to  six  months  delay  _ 3.  Seven  to  twelve  month  delay  _ 4.  Over  a  year  late 

5.  Did  not  reach  production,  was  not  implemented  _ 8.  Not  Applicable  _ 9.  Don’t  know 

06.  After  the  SYSTEM  was  actually  in  Production,  how  many  additional  changes  in  designs  and  processes  were  required? 

_  1.  Many  serious  changes  X  2.  Significant  changes  _ 3.  Minor  changes  _ 4.  No  or  almost  no  changes 

7.  System  did  not  reach  production  _ 8.  Not  Applicable  _ 9.  Don’t  know 

07.  Did  the  SYSTEM  as  it  was  implemented  meet  the  program’s  cost  goals? 

_ 1.  The  results  met  or  exceeded  cost  goals  7.  System  did  not  reach  production. 

X  2.  The  results  came  close  to  achieving  cost  goals  8.  Not  applicable. 

_  3.  The  results  fell  far  short  of  achieving  cost  goals  9.  Don’t  know. 

08.  Did  the  System  Development  program,  as  implemented,  come  in  on  budget? 

_  1 .  The  project  met  or  under-ran  budget.  _ 9.  Don’t  know. 

_ 2.  The  project  slightly  exceeded  budget. 

X  3.  The  project  significantly  exceeded  budget. 

09.  Did  the  System  as  it  was  implemented  meet  the  project’s  technical  goals  and  functional  requirements? 

X  1 .  The  results  met  or  exceeded  technical  goals  _ 7.  Did  not  reach  production,  was  not  implemented. 

_  2.  The  results  came  close  to  achieving  technical  goals  _ 8.  Not  applicable. 

_  3.  The  results  fell  far  short  of  achieving  technical  goals  _ 9.  Don’t  know. 

09.  Did  the  System  have  problems  in  the  field  under  operational  conditions  in  Desert  Storm? 

_  1.  Yes,  problems  in  the  field  significantly  limited  the  system's  effectiveness. 

X  2.  Yes,  problems  in  the  field  caused  minor  problems  in  the  system’s  effectiveness.  _ 9.  Don’t  know,  question 

_  3.  No,  the  system  was  deployed  and  encountered  no  noticeable  loss  of  effectiveness,  not  applicable  to  this  system. 

_ 4.  No,  the  system  was  deployed  and  exceeded  expectations  of  its  effectiveness. 

IF  YOU  CHECKED  “1”  or  “2”  to  question  09,  what  did  the  field  problems  result  from?  Check  all  that  apply. 

09a. _ The  System  did  not  meet  its  requirements. 

09b.  X _ Requirements  did  not  reflect  the  field  environment.  09e. _ Other: _ 

09c. _ The  System  was  not  deployed  in  its  intended  role.  (Please  explain.) _ 

09d. _ Personnel  not  adequately  trained/prepared  to  use  the  System. 
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11.  Now  that  you  have  had  a  chance  to  think  about  the  project  and  provide  some  answers,  how  well  do 
you  think  you  feel  you  have  captured  the  details  of  the  project?  Are  you:  (Check  S  one.) 

X _ 1.  Very  confident  that  you  captured  the  project  well? 

_ 2.  Fairly  confident  you  understand  the  main  things  well,  but  not  as  confident  about  the  details? 

_ 3.  Not  confident  of  your  information  about  the  project,  so  we  should  only  use  your  answers  with 

caution. 

12.  Finally,  what  was  the  most  difficult  problem  the  Project  Manager  faced,  how  was  the  problem  dealt 
with,  and  what  was  the  impact  of  the  problem  on  the  project  outcome? 

Funding  Issues 
Production  Control 


Technology  Readiness  Level  Scale 

1.  Basic  principles  observed  and  reported.  Scientific  research  begins  to  be  translated  into  applied 
research  and  development  concepts.  There  have  been  paper  studies  of  technology’s  basic  properties. 

2.  Technology  concept  and/or  application  formulated.  Practical  applications  have  been  invented. 
Application  is  specu¬ 
lative  and  there  is  no  proof  or  detailed  analysis  to  support  the  assumptions.  Examples  are  still  limited 
to  paper  studies. 

3.  Analytical  &  experimental  critical  function  and/or  characteristic  proof  of  concept.  Analytical 
and  laboratory 

studies  have  physically  validated  analytic  predictions  of  separate  elements  of  the  technology. 
Examples  include  components  that  are  not  yet  integrated  or  representative 

4.  Component  and/or  bread  board  validation  in  lab  environment.  Basic  technological  components 
are  integrated  to  establish  that  pieces  will  work  together,  e.g.,  integration  of  ad  hoc  parts  in  lab.  This 
is  relatively  “low  fidelity”  compared  to  the  eventual  system. 

5.  Components  and/or  bread  board  validation  in  relevant  environment.  Fidelity  of  breadboard 
technology  is  significantly  increased.  Basic  components  integrated  with  reasonably  realistic 
supporting  elements  so  the  technology  can  be  tested  in  a  simulated  environment.  Examples  include 
“high  fidelity”  laboratory  integration  of  components. 

6.  System/subsystem  model  or  prototype  demonstrated  in  a  relevant  environment.  Representative 
model  or  prototype  system,  which  is  well  beyond  the  breadboard  tested  for  TRL  5,  tested  in  a 
relevant  environment.  Represents  a  major  step  up  in  a  technology’s  demonstrated  readiness. 

Examples  include  testing  a  prototype  in  a  high  fidelity  laboratory  environment  or  in  a  simulated 
operational  environment. 

7.  System  prototype  demonstrated  in  an  operational  environment.  Prototype  near  or  at  planned 
operational  system.  Represents  a  major  step  up  from  TRL  6,  requiring  the  demonstration  of  an  actual 
system  prototype  in  an  operational  environment,  such  as  in  an  aircraft,  vehicle  or  space.  Examples 
include  testing  the  prototype  in  a  test  bed  aircraft. 

8.  Actual  system  completed  and  qualified  in  test  and  demonstration.  Technology  proven  to  work  in 
final  form  and  under  expected  conditions.  In  almost  all  cases,  this  TRL  represents  the  end  of  true 
system  development.  Examples  include  developmental  test  and  evaluation  in  its  intended  weapon 
system  to  determine  if  it  meets  design  specification. 

9.  Actual  system  proven  in  successful  operational  environment.  Actual  application  of  the  technology  in  its  final 
form  and  under  mission  conditions,  such  as  those  encountered  in  operational  test  and  evaluation.  In  almost  all 
cases,  this  is  the  end  of  the  last  “bug  fixing”  aspects  of  true  system  development.  Examples  include  using  the 
system  under  operational  mission  conditions. 
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APPENDIX  D:  ACRONYMS  AND  ABBREVIATIONS 


AAH 

Advanced  Attack  Helicopter  or  Apache  Attack  Helicopter 

AAFSS 

Advanced  Aerial  Fire  Support  System 

CAIG 

Cost  Analysis  Improvement  Group 

CAS 

Close  Air  Support 

D 

Development  (survey  definition) 

DoD 

Department  of  Defense 

FMECA 

Failure  Modes  Effects  and  Criticality  Analysis 

FUE 

First  Unit  Equipped 

IPT 

Integrated  Product  Team 

IR&D 

Internal  Research  and  Development 

MNS 

Mission  Needs  Statement 

PM 

Program  Manager 

PMO 

Program  Management  Office 

RFP 

Request  for  Proposal 

SP 

System  planning  phase  (survey  definition) 

S&T 

Science  and  Technology 

TEMP 

Test  Evaluation  Master  Plan 

TP 

Transition  to  Production  (survey  definition) 

TRL 

Technology  Readiness  Level 
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